tag:status.ix.asn.au,2005:/historyInternet Association of Australia Status - Incident History2024-03-19T02:29:54ZInternet Association of Australiatag:status.ix.asn.au,2005:Incident/201720192024-04-05T23:00:00Z2024-03-06T02:43:11Z[MAINTENANCE] ACT-IX | Inter-DC connectivity maintenance<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>5</var>, <var data-var='time'>23:00</var> UTC - Apr <var data-var='date'>6</var>, <var data-var='time'>03:00</var> UTC</strong></p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>02:43</var> UTC</small><br><strong>Scheduled</strong> - Our connectivity provider in Canberra will be performing maintenance on our link between NEXTDC C1 and TransACT DC. Members with services on ACT-IX should expect a loss of connectivity to peers not located in the same POP during the maintenance window.<br /><br />Date: 06 April 2024<br />Start time: 10:00 AM (AEDT)<br />Downtime to expect: ~60 minutes<br />Window: 4 hours<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/187371642023-10-25T14:59:34Z2023-10-25T14:59:35Z[MAINTENANCE] NSW-IX | VIC-IX | QLD-IX | SA-IX | Route Server 2 Migrations<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>14:59</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed. If you have any questions or concerns, please contact us on 1300 793 320 or support@internet.asn.au.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>23:21</var> UTC</small><br><strong>Scheduled</strong> - IAA engineers will be migrating Route Server 2 on NSW-IX, VIC-IX, QLD-IX and SA-IX to new hardware.<br /><br />ALL BGP sessions to Route Server 2 on NSW-IX, VIC-IX, QLD-IX and SA-IX will go down briefly during the migration. Please ensure you have configured BGP sessions to Route Server 1 on each respective IX for redundancy.<br /><br />Affected peer IPs:<br />rs2.nsw - 218.100.52.254<br />rs2.vic - 218.100.78.254<br />rs2.qld - 218.100.76.254<br />rs2.sa - 218.100.54.126<br /><br />Date: 26/Oct/2023<br />Start time: 12:00 midnight (AEST)<br />Downtime to expect: Up to 30 minutes<br />Window: 4 hours<br /><br />The work is expected to take approximately 30 minutes per IX. However, 4 hours have been booked in the unlikely event of an emergency.<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or support@internet.asn.au.</p>tag:status.ix.asn.au,2005:Incident/187371612023-10-24T16:02:44Z2023-10-24T16:02:44Z[MAINTENANCE] NSW-IX | VIC-IX | QLD-IX | SA-IX | Route Server 1 Migrations<p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>16:02</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed. If you have any questions or concerns, please contact us on 1300 793 320 or support@internet.asn.au.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>23:20</var> UTC</small><br><strong>Scheduled</strong> - IAA engineers will be migrating Route Server 1 on NSW-IX, VIC-IX, QLD-IX and SA-IX to new hardware.<br /><br />ALL BGP sessions to Route Server 1 on NSW-IX, VIC-IX, QLD-IX and SA-IX will go down briefly during the migration. Please ensure you have configured BGP sessions to Route Server 2 on each respective IX for redundancy.<br /><br />Affected peer IPs:<br />rs1.nsw - 218.100.52.1<br />rs1.vic - 218.100.78.1<br />rs1.qld - 218.100.76.1<br />rs1.sa - 218.100.54.1<br /><br />Date: 25/Oct/2023<br />Start time: 12:00 midnight (AEST)<br />Downtime to expect: Up to 30 minutes<br />Window: 4 hours<br /><br />The work is expected to take approximately 30 minutes per IX. However, 4 hours have been booked in the unlikely event of an emergency.<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or support@internet.asn.au.</p>tag:status.ix.asn.au,2005:Incident/188942342023-10-23T19:12:39Z2023-10-23T19:12:39Z[Emergency Maintenance] SA-IX | YourDC Edinburgh Parks | pe1.adl3 reboot<p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>19:12</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Update</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>03:58</var> UTC</small><br><strong>Scheduled</strong> - IAA engineers have identified a fault condition with the pe1.adl3 switch at YourDC Edinburgh Parks, requiring a reboot of the switch to resolve. Engineers will be conducting this switch reboot at 05:30 AM ACDT on Tuesday 24th October. All services in YourDC Edinburgh Parks (adl3) as well as all BGP sessions to SA-IX Route Server 1 (rs1.sa) will go offline briefly as the switch is rebooted. Ports will be drained of peering traffic using BCP214-compliant filters before they are migrated. Filters will be removed once the ports are back online and stable.<br /><br />Date: 24/Oct/2023<br />Start time: 05:30 AM (ACDT)<br />Downtime to expect: Up to 10 minutes<br />Window: 4 hours<br /><br />The work is expected to take approximately 10 minutes. However, 4 hours have been booked in the unlikely event of an emergency.<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or support@internet.asn.au.</p>tag:status.ix.asn.au,2005:Incident/188466272023-10-19T14:27:11Z2023-10-19T14:27:12Z[Emergency Maintenance] SA-IX | YourDC Hawthorn | pe1.adl4 reboot<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>14:27</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>23:53</var> UTC</small><br><strong>Scheduled</strong> - IAA engineers have identified a fault condition with the pe1.adl4 switch at YourDC Hawthorn, requiring a reboot of the switch to resolve. Engineers will be conducting this switch reboot at 12:30 AM ACDT on Friday 20th October. All services in YourDC Hawthorn (adl4) will go offline briefly as the switch is rebooted.<br /><br />Date: 20/Oct/2023<br />Start time: 12:30 AM (ACDT)<br />Downtime to expect: Up to 15 minutes<br />Window: 2 hours<br /><br />The work is expected to take approximately 15 minutes. However, 2 hours have been booked in the unlikely event of an emergency.<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or support@internet.asn.au.</p>tag:status.ix.asn.au,2005:Incident/186735862023-10-18T23:45:46Z2023-10-18T23:45:46Z[MAINTENANCE] WA-IX | IPv6 BGP Session Migrations<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>23:45</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>00:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>03:43</var> UTC</small><br><strong>Scheduled</strong> - IAA engineers will be migrating IPv6 route server BGP sessions to our new route servers on WA-IX. During the migrations, members with IPv6 BGP sessions on WA-IX will see their BGP sessions go down briefly while they are migrated.<br /><br />The migrations will be completed in stages: Sessions to rs1.wa will be migrated first, followed by sessions to rs2.wa. Engineers will ensure that at least one IPv6-capable route server is available on WA-IX at all times during the migrations.<br /><br />Date: 18/Oct/2023<br />Start time: 08:00 AM (AWST)<br />Downtime to expect: Up to 15 minutes per IPv6 BGP session<br />Window: 4 hours<br /><br />The work is expected to take approximately 2 hours in total. However, 4 hours have been booked in the unlikely event of an emergency.<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or support@internet.asn.au.</p>tag:status.ix.asn.au,2005:Incident/186739022023-10-03T06:09:15Z2023-10-03T06:09:15Z[OUTAGE] SA-IX | Power loss to rs1.sa<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>06:09</var> UTC</small><br><strong>Resolved</strong> - Power was restored roughly ~3:53pm (ACDT), few minor issues with restoring RS1.SA but has come up all good now and peers established. <br /><br />DC operator YourDC Ed Park confirmed this was related to a power event at the site impacting a power feed to our rack (Power Room 1 Offline). <br /><br />Unfortunately RS1.SA is a single PSU device which is set to be upgraded to a dual PSU device very soon to avoid issues like this, not soon enough it seems!</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>04:31</var> UTC</small><br><strong>Investigating</strong> - IAA engineers are aware of a power issue that has taken the rs1.sa route server on SA-IX offline. Engineers have engaged the data centre operator to investigate, and further updates will be provided as information comes to hand.</p>tag:status.ix.asn.au,2005:Incident/184963502023-09-15T04:15:17Z2023-09-15T04:15:17Z[MAINTENANCE] ACT-IX | RS2.ACT | NextDC C1 B-Feed Power Maintenance<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>04:15</var> UTC</small><br><strong>Completed</strong> - NextDC has postponed this maintenance, a new notification will be sent out once its been rescheduled.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>02:43</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers have been notified of scheduled power works occurring at NextDC C1 that will impact ACT-IX Route Server 2.<br /><br />Window: Thursday September 21, 2023 between the hours of 6:00 and 18:00 (AEST)<br /><br />Impact Summary: <br />* Route Server 2 (rs2.act.ix.asn.au) WILL be impacted by these power works as it utilises a non-redundant power supply, members should expect BGP to drop toward this route server for the maintenance window. <br />v4: 218.100.54.254/25<br />v6: 2001:7fa:11:5:0:1db6:0:2/64<br /><br />To ensure this is non-service impacting, ACT-IX members are advised to verify they have established BGP sessions with Route Server 1 (rs1.act.ix.asn.au):<br />v4: 218.100.54.129/25<br />v6: 2001:7fa:11:5:0:1db6:0:1/64<br /><br /><br />If you have any questions or concerns regarding this notice please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/184287872023-09-08T07:12:30Z2023-09-08T07:12:30Z[OUTAGE] Portal issues<p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>07:12</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>03:18</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>00:58</var> UTC</small><br><strong>Identified</strong> - IAA member portal - portal.internet.asn.au - is experiencing an outage and a fix is being worked on. At this time member provisioning operations are on hold until this is resolved.</p>tag:status.ix.asn.au,2005:Incident/180378312023-08-03T05:33:50Z2023-08-03T05:33:50Z[OUTAGE] National | Route Server 2 Session Flap<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>05:33</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>04:54</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>04:34</var> UTC</small><br><strong>Update</strong> - Route server config generator change has been reverted for internal review.<br /><br />Route server 2 nationally will now be re-deployed to the previous configuration to resolve this - we expect the same sessions to flap once more but no further issues after that.<br /><br />Apologies for the inconvenience!</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>04:10</var> UTC</small><br><strong>Identified</strong> - Route Server 2 peers across each IX unexpectedly flapped after a configuration generator change, investigations are underway (automation disabled whilst its looked into).</p>tag:status.ix.asn.au,2005:Incident/175548602023-06-27T14:17:51Z2023-06-27T14:17:52Z[MAINTENANCE] VIC-IX | VDC-MEL03 | MDC/5GN Re-termination<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>14:17</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>13:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>03:09</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers will be finishing off our rack migration at Vocus VDC-MEL03 (mel5) by re-terminating member ports connecting at level 15 MDC/5GN via a temporary cable down to level 3 to our own cable. <br /><br />Date: 27/06/2023<br />Start time: 11:00pm (AEST)<br />Downtime to expect: ~15-30 minutes<br />Window: 2 hours <br /><br />* Members with service ports connected to VDC-MEL03 via MDC/5GN on level 15 should expect their port to go down during this window, affecting all services associated with the port: Peering, VLL, Extended Reach Peering and any bilateral connections you may have to another peer.<br /><br />Ports will be drained of peering traffic using BCP214 compliant filters before they are migrated. Filters will be removed once the ports are back online and stable.<br /><br /><br />The work is expected to take about 1 hour.<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/176850622023-06-26T13:06:34Z2023-06-26T13:06:35Z[OUTAGE] PER-ADL | Path Outage<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>13:06</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>06:57</var> UTC</small><br><strong>Identified</strong> - We have lost our primary and secondary paths between Perth <> Adelaide. This will impact inter-capital VLLs and Extended Reach Peering services traversing this path. <br /><br />Path #1 went down this morning 8:46am (AEST) and the carrier repairs are underway. No ETA. <br />Path #2 started flapping causing customer impacts. Brief flap at: 2:14pm and 2:46pm (AEST), and a more disruptive link down/up at 4:39pm.<br /><br />Path #2 is back up however the stability is unclear, this status notice will remain open until we can confirm its stable or path #1 restores.</p>tag:status.ix.asn.au,2005:Incident/174978972023-06-14T12:00:07Z2023-06-14T12:00:07Z[MAINTENANCE] Member Portal | IRRd server upgrades<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>07:10</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>07:10</var> UTC</small><br><strong>Scheduled</strong> - IAA engineers will be conducting upgrades on the server hosting our IRRd instance. During the upgrades, members will be unable to order new peering services via the member portal. No other portal functions will be affected.<br /><br />Date: 14/06/2023<br />Start time: 8:00pm (AEST)<br />Downtime to expect: Up to 2 hours<br />Window: 4 hours<br /><br />The work is expected to take approximately 2 hours. However, 4 hours have been booked in the unlikely event of an emergency.<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or support@internet.asn.au.</p>tag:status.ix.asn.au,2005:Incident/168871072023-05-10T15:23:54Z2023-05-10T15:23:56Z[MAINTENANCE] VIC-IX | VDC-MEL03 | Rack Migration<p><small>May <var data-var='date'>10</var>, <var data-var='time'>15:23</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>17</var>, <var data-var='time'>04:58</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers will be migrating racks at Vocus VDC-MEL03 (mel5) as well as upgrading site hardware to support 40/100G ports, this migration will cause an outage on all services delivered to that site. <br /><br />Date: 11/05/2023<br />Start time: 12:00am (AEST)<br />Downtime to expect: Up to 1 hour<br />Window: 4 hours <br /><br />* Members with service ports connected to VDC-MEL03 should expect their port to go down during this window, affecting all services associated with the port: Peering, VLL, Extended Reach Peering and any bilateral connections you may have to another peer.<br /><br />Ports will be drained of peering traffic using BCP214 compliant filters before they are migrated. Filters will be removed once the ports are back online and stable.<br /><br /><br />The work is expected to take about 2 hours. However, 4 hours have been booked in the unlikely event of an emergency. <br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/169682012023-05-08T15:00:11Z2023-05-08T17:40:57Z[MAINTENANCE] VIC-IX | Equinix ME1 | pe1.mel4, RS2.VIC | Service Migrations<p><small>May <var data-var='date'> 8</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>03:33</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers will be removing legacy hardware at Equinix ME1 (mel4) which includes migrating peers over to our modern switch, this migration will cause an outage on all peers connected to pe1.mel4 which also includes the RS2.VIC uplink. pe2.mel4 is unaffected. <br /><br />Date: 09/05/2023<br />Start time: 12:00am (AEST)<br />Downtime to expect: Up to ~15 minutes<br />Window: 3 hours <br /><br />* ALL VIC-IX members should expect an interruption toward RS2.VIC whilst its uplink is migrated. <br /><br />* Members with service ports connected to pe1.mel4 should expect their port to go down during this window, affecting all services associated with the port: Peering, VLL, Extended Reach Peering and any bilateral connections you may have to another peer.<br /><br />Ports will be drained of peering traffic using BCP214 compliant filters before they are migrated. Filters will be removed once the ports are back online and stable.<br /><br />The work is expected to take about 1 hour. However, 3 hours have been booked in the unlikely event of an emergency. <br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/168465832023-04-20T15:11:59Z2023-04-20T15:12:00Z[MAINTENANCE] SA-IX | AS10084 Content Network | Hardware Upgrade<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>15:11</var> UTC</small><br><strong>Completed</strong> - Hardware issues were encountered, and in the interest of time this has been rolled back with another maintenance window to come.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>13:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>06:32</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers will be upgrading our AS10084 routing hardware which will cause an outage on content services for SA-IX (includes Netflix and GGC). <br /><br />Start time: 11:00pm (ACST)<br />Expected downtime: Up to 1 hour <br />Window: 4 hours <br /><br />The work is expected to take about 1 hour. However, 4 hours have been booked in the unlikely event of an emergency. <br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/165030012023-04-10T18:01:04Z2023-04-10T18:01:06Z[MAINTENANCE] VIC-IX | QLD-IX | Firmware Upgrades and IGP Changes<p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>18:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>01:57</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers will be upgrading the firmware of the switching hardware across VIC-IX and QLD-IX to resolve several software bugs and continue flattening the OSPF domain.<br /><br />Window: 4 hours <br />Expected Downtime: ~15 minutes per site <br />Impacted sites:<br />VIC-IX --- NextDC M1, Equinix ME1 (Excludes: VDC-MEL03)<br />QLD-IX --- NextDC B1, NextDC B2, Host Networks<br /><br /><br />* Members with service ports at the above sites should expect their port to go offline for up to ~10 minutes whilst each switch is being upgraded. <br /><br />Service ports will be drained of peering traffic roughly 5 minutes before their respective upgrades using BCP214 compliant filters. Filters will be removed once the devices are back online and stable.<br /><br />* We will ensure at least one (1) route server is up at ALL times during the window. Please verify you are peering with both route servers to minimise the impact:<br /><br />VIC-IX<br />RS1 - 218.100.78.1/24 & 2001:7fa:11:1:0:1db6:0:1/64<br />RS2 - 218.100.78.254/24 & 2001:7fa:11:1:0:1db6:0:2/64<br /><br />QLD-IX<br />RS1 - 218.100.76.1/24 & 2001:7fa:11:2:0:1db6:0:1/64<br />RS2 - 218.100.76.254/24 & 2001:7fa:11:2:0:1db6:0:2/64<br /><br />The work is expected to take about 1 hour per exchange. However, 4 hours have been booked in the unlikely event of an emergency. <br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/167617522023-04-05T19:59:48Z2023-04-05T19:59:49Z[OUTAGE] NSW-IX RS2 issues<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>19:59</var> UTC</small><br><strong>Resolved</strong> - The issue was found and resolved during troubleshooting, which also necessitated the reboot of several switches. We apologise for any inconvenience caused, and appreciate members' patience as we worked to resolve this fault condition.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>17:25</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue with reachability to RS2 on NSW-IX. We will be disabling BGP to RS2 until the issue is resolved.</p>tag:status.ix.asn.au,2005:Incident/165036962023-04-05T15:47:34Z2023-04-05T15:47:35Z[MAINTENANCE] NSW-IX | Firmware Upgrades and IGP Changes<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>15:47</var> UTC</small><br><strong>Completed</strong> - All scheduled maintenance works have now been completed. We regret that due to an oversight stemming from the recent rollback of DST in the Sydney timezone, this maintenance commenced an hour earlier than previously communicated. We apologise for any inconvenience caused by this.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>02:55</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers will be upgrading the firmware of the switching hardware across NSW-IX to resolve several software bugs and continue flattening the OSPF domain. <br /><br />Window: 4 hours <br />Expected Downtime: ~15 minutes per site <br /><br />* Members with service ports across NSW-IX should expect their port to go offline for up to ~10 minutes whilst each switch is being upgraded. Please note a few legacy devices will not experience any outage (NextDC S1 - pe1.syd4, Equinix SY3 - pe1.syd3, Equinix SY1 - pe4.syd1). <br /><br />Service ports will be drained of peering traffic roughly 5 minutes before their respective upgrades using BCP214 compliant filters. Filters will be removed once the devices are back online and stable.<br /><br />* We will ensure at least one (1) route server is up at ALL times during the window however downtime is not expected due to their active/standby uplink fail over. With that said please verify you are peering with both route servers to minimise any potential impact:<br /><br />NSW-IX<br />RS1 - 218.100.52.1/23 & 2001:7fa:11:4:0:1db6:0:1/64<br />RS2 - 218.100.52.254/23 & 2001:7fa:11:4:0:1db6:0:2/64<br /><br /><br />The work is expected to take about 1-2 hours. However, 4 hours have been booked in the unlikely event of an emergency.<br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/164111162023-03-27T14:55:54Z2023-03-27T14:55:55Z[MAINTENANCE] SA-IX | ACT-IX | IGP Changes and Reboot<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>14:55</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>05:42</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers will be flattening the OSPF domain at SA-IX and ACT-IX as well as rebooting hardware at VocusDC/132 Franklin and AdamDC to resolve an outstanding issue.<br /><br />All sites within SA-IX and ACT-IX will be under hazard notice as the changes are implemented, no outage is expected. The only downtime is expected at VocusDC and AdamDC within SA-IX. <br /><br />Window: 3 hours <br />Expected Downtime: ~15 minutes<br />Sites: <br />SA-IX --- VocusDC, YourDC Hawthorn, YourDC Ed Park, Colocity DC3, AdamDC<br />ACT-IX --- NextDC C1, TransACT<br /><br />* Members with service ports at Vocus DC and AdamDC (SA-IX) should expect their port to go offline for up to ~10 minutes whilst each switch is being rebooted. <br /><br />Service ports will be drained of peering traffic roughly 5 minutes before their respective reboots using BCP214 compliant filters. Filters will be removed once the devices are back online and stable.<br /><br />* All members at other sites listed should not expect any outage as traffic will re-route with the changes.<br /><br />* AS10084 Content network will be impacted by the reboot at VocusDC (SA-IX) and all SA-IX members should expect an interruption in content being served. <br /><br />The work is expected to take about 1 hour. However, 3 hours have been booked in the unlikely event of an emergency. <br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/165691092023-03-20T06:09:27Z2023-03-20T06:09:27Z[OUTAGE] SA to/from NSW + QLD | Adjacency failure<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>06:09</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>04:10</var> UTC</small><br><strong>Monitoring</strong> - An issue triggered at ~2:27pm (AEDT) following a flap on our Sydney <> Adelaide direct path that caused impacts on Extended Reach Peering (ERP) services as well as Virtual Leased Lines (VLL) between Adelaide and NSW + QLD. Adelaide to WA + VIC + ACT were unaffected. <br /><br />This issue is isolated to a suspected race condition with a fix applied ~2:47pm (AEDT) and the root cause is under investigation.</p>tag:status.ix.asn.au,2005:Incident/163116872023-03-14T20:49:16Z2023-03-14T20:49:17Z[MAINTENANCE] WA-IX | QV1, Equinix PE2 | Firmware Upgrades and IGP Changes<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>20:49</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>06:20</var> UTC</small><br><strong>Scheduled</strong> - IAA Engineers will be upgrading the firmware of the remaining switching hardware across WA-IX to resolve several software bugs and additionally begin flattening the OSPF domain with other states to follow. <br /><br />Window: 3 hours <br />Expected Downtime: ~15 minutes per site <br />Impacted sites: QV1, Equinix PE2 <br /><br />* Members with service ports at the above sites should expect their port to go offline for up to ~10 minutes whilst each switch is being upgraded. <br /><br />Service ports will be drained of peering traffic roughly 5 minutes before their respective upgrades using BCP214 compliant filters. Filters will be removed once the devices are back online and stable.<br /><br />* We will ensure at least one (1) route server is up at ALL times during the window. Please verify you are peering with both route servers to minimise the impact:<br /><br />RS1 - 198.32.212.1/24 & 2001:7fa:11::1db6:0:1/64 <br />RS2 - 198.32.212.254/24 & 2001:7fa:11::1db6:0:2/64 <br /><br /><br />The work is expected to take about 1 hour. However, 3 hours have been booked in the unlikely event of an emergency. <br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/164628822023-03-13T09:43:11Z2023-03-13T09:43:12Z[OUTAGE] VIC-IX | VDC-MEL03/ MDC | Rack Power Loss<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>09:43</var> UTC</small><br><strong>Resolved</strong> - Our secondary power rail has restored following operator maintenance. This incident is now resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>23:27</var> UTC</small><br><strong>Identified</strong> - Power dropped again to the site and is currently back up on partial power. Power drop may occur again. DC operator electricians working onsite to resolve.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>22:46</var> UTC</small><br><strong>Monitoring</strong> - Power was partially restored ~9:24am (AEDT) which has brought member services back up. Pending full power re-establishment.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>22:09</var> UTC</small><br><strong>Identified</strong> - Confirmed the site is having power issues and is offline, updates to follow.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>22:08</var> UTC</small><br><strong>Investigating</strong> - We have lost power to our devices at our rack in VDC-MEL03, investigations are underway.</p>tag:status.ix.asn.au,2005:Incident/163842452023-03-08T20:18:24Z2023-03-08T20:18:24Z[EMERGENCY MAINTENANCE] WA-IX | VDC-PER01 / PerthIX (per2) | I2C protocol panic<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>20:18</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>03:45</var> UTC</small><br><strong>Scheduled</strong> - IAA engineers have received monitoring alerts of our switch at VDC-PER01 / PerthIX experiencing an I2C protocol panic causing system reporting instability. No traffic impact has occurred or is expected.<br /><br />Our technician has attended onsite and after re-seating various redundant bits of hardware in an attempt to restore the stuck I2C communications, the issue persists and the best next course of action is to reboot the device.<br /><br />Expected downtime: ~15 minutes <br />If hardware fault: ~1-3 hours<br /><br />Ports will be drained of peering traffic using BCP214 compliant filters before the reboot and removed once the device is back online and stable.<br /><br />In the unlikely event of a hardware fault, a spare onsite switch will be swapped out with the failed hardware. <br /><br />If you have any questions or concerns regarding this notice, please contact us on 1300 793 320 or noc@ix.asn.au.</p>tag:status.ix.asn.au,2005:Incident/163493362023-03-06T10:22:27Z2023-03-06T10:22:27Z[OUTAGE] East to West | Path Outage<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>01:28</var> UTC</small><br><strong>Monitoring</strong> - Our Adelaide to Melbourne path revived at ~12:11pm AEDT restoring services . Our Adelaide to Sydney path is still under an unplanned outage and currently re-routing via Melbourne.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>00:47</var> UTC</small><br><strong>Investigating</strong> - We have lost primary and secondary paths between Adelaide and Melbourne + Sydney which will impact inter-capital VLLs. Investigations are underway.</p>