[Emergency Maintenance] NSW-IX | Stuck BCP214 ACL
Scheduled Maintenance Report for Internet Association of Australia
Completed
The scheduled maintenance has been completed.
Posted Feb 12, 2022 - 02:32 AEDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Feb 12, 2022 - 01:00 AEDT
Scheduled
IAA engineers have confirmed a bug impacting a number of peers across NSW-IX whereby previous maintenance's utilising ingress BCP214 access-lists are now stuck in hardware, causing these peers to be unable to send BGP open messages to destination port 179 - however can still establish BGP via their dynamic reply port.

We have decided to initiate an emergency outage across all of NSW-IX tonight to resolve this issue permanently, we have already conducted an emergency reboot and firmware upgrade on a switch impacted and confirmed its resolved.


Member ports within each datacentre will be drained of peering traffic using BCP214 compliant filters and removed once the switching hardware is back online and stable.

Expected Downtime: ~15 minutes per site


Be mindful that you may observe similar connectivity downtime to one of the route servers before/after your port downtime is experienced as other sites are upgraded. Please note we will always ensure at least one route server is online at a time and sites verified before proceeding with the next.

Member ports with Inter-capital VLL services may also experience higher latency while sites are upgraded as traffic re-routes via alternate paths.


IAA has spare hardware on standby in the unlikely event a device requires replacing, thus a 4 hour window has been set.

We apologise for the short notice and any inconvenience this may cause.

Please contact peering@ix.asn.au or 1300 793 320 should you have any queries regarding this notice.
Posted Feb 11, 2022 - 14:47 AEDT
This scheduled maintenance affected: Peering Fabric (NSW-IX) and Inter-Capital Network (MEL-SYD, SYD-BNE, SYD-ACT, SYD-ADL).