Fictional stories about the network: how I taught physics from my own and others' mistakes
In practice, I have repeatedly convinced that physics is stubborn and will necessarily avenge attempts to neglect its laws. For errors in network physics, I paid quarterly bonuses, fixing jambs at night, and users любов love ’. But such a school of life is remembered once and for all.
Today I want to share a selection of stories about the physics of networks and formulate the rules of network life, which I put into practice.
Disclaimer: the article contains stories from my experience in large and small enterprises and telecom operators. Many of them happened to me or colleagues at the dawn of a career. Most characters are collective images, any coincidences are random. My opinion may not coincide with the opinion of DataLine.
# 1. The fate of the enterprise hangs on a patch cord
A typical day, nothing portends trouble, except for the Very-Important-Meeting of top managers of the company. The technical service has the latest request for today: you need to connect a new user.
Valorous tech support
15 minutes to an important meeting.
As luck would have it, the organizer is crowded. Need to free up space and "reconnect" some wires. We clear the clearing, insert everything back.
One awkward move - and that very important patch cord to the conference system was outside. That's bad luck, the connector tongue was torn or the connector for some reason was already broken.
Before an important meeting in the same conference system - 10 minutes.
Without thinking twice, the tech support warrior inserts the broken patch cord back into the switch. In principle, everything holds, everything works.
... At a very important meeting, the fate of the enterprise is decided: will employees have an annual bonus and other important points.
During the performance of the Most Important Top Manager, the broken patch cord treacherously starts moving and falls out of the switch connector. The main event of the year remains unconnected.
What was the reaction of the participants, it is not difficult to guess.
Patch Cord Rules:
- I always use ready-made patch cords with castle protection - and no homemade. If the tongue is broken, it’s better not to be lazy and replace the patch cord.
- If you still have to use the self-propeller as a temporary solution, I use a high-quality multicore cable (stranded or patch) and protective caps.
- I do not advise to carry out work if an important task is planned, where it is better not to touch anything. As a last resort, you need to clearly understand what you are doing. There is also a well-known sign that work on Friday, especially in the afternoon, is for a "fun" weekend.
# 2. The Little Lead
The procurement department of the telecom company learned about the price advantages of copper-plated twisted pair from aluminum. Cheap and cheerful, take it! It takes a lot of wires, so the buyers took a large batch, saved the company's budget and received their bonus for this.
9 months have passed. Immediately a bunch of applications began to fly in: something is missing, everything flops, CRC errors on ports grow. Where is my internet and what am I paying for?
The purchasers did not know that aluminum and copper form a galvanic pair. For 9 months, our “copper-plated” has experienced many temperature differences, and the connectors have oxidized. But this is still half the pain. Almost all of the twisted pair segments were over 40 meters. Not only are the connectors oxidized, but the link also jumps due to the large length of the copper plated.
Therefore, two winter months out of three, the installers traveled to the points of presence of the telecom operator and shifted the lines, instead of connecting new customers. Re-crimping of the connectors continued until they switched to pure copper.
Twisted Pair Rules:
- I’m never using “copper” (or “poor”, as I call it). The main advantage is the price, and this is where all the pros end.
If you turn on PoE power, you can get unpredictable effects, up to the equipment failure. As a rule, "copper-plated" is a two-park, so you can’t see speeds of more than 100 megabits. The temperature and humidity in the cabinets can be different, the contacts oxidize very quickly, and the links fall by 10 megabytes.
- If the household is inherited, it is very simple to check the twisted pair. If the copper-plated is steel - with a magnet. If aluminum - look at the end of the connector or remove the cable from the bay and look at the markings.
# 3. My shorty friend
There is a point in any connection instructions: make sure that there is no voltage in the low voltage. The administrator went to connect a new user, but heard the usual “faster-faster” over his ear and connected without checking.
Blinked once, and the POE panel turned off.
The second blinked, and half of the office was left without communication - employees' phones were powered through the PoE-panel.
The third blinked, and all were left without applications - users' computers were also connected by telephone.
At this time, the connected employee puzzled bent over the power supply. As soon as the computer was turned on, you could hear the sound of the good old dialup modem that was publishing the power supply. The device did not burn out completely and gave us a reason to nostalgia.
The rest of the employees were more fortunate: the panel had short-circuit protection. It was enough to turn it off and on, and, lo and behold, it worked. The IT department escaped with a slight startle.
Before connecting, I check the voltage on the patch cord with a tester with PoE support. It’s good practice to check the tester with all the channels in general before turning them on. Looking at the socket and switch port is not enough. While you go to the outlet on the other hand, things can change.
# 4. Fashionable * business center
One day, our office moved to a new business center. Time passed, gigabit was already missing, LACP did not drag, it was necessary to switch to a 10G network. They started to lift.
Some channels rose on the top ten at once, some did not rise at all. One rose somehow strangely: there were too many errors on the ports. The project risked not being completed on time, and I began to dig.
There was no marking on the cable in the BC. I had to climb to the very ceiling and find cables with markings. It turned out that the connection was organized as follows: on the one hand there are multimode patch cords, then a single-mode fiber, and on the other side - again multimode patch cords.
As we know, single mode and multimode differ not only in diameter but also in wavelength. This situation turned out:
Representatives of the business center had to admit the error and digest all the cables.
* either single-mode or multi-mode
- I do not believe what is written on paper and I check everything myself. I check the types of optical cables, but the rule applies not only to optics.
- I use FOCL of the required OM class (2-5) and do not mix different OM. Otherwise, you need to be ready to work on the decline.
- Before receiving FOCL, I get the OTDR test results from specialists who cooked couplings and cross.
# 5. More about optics and patch cords
The difference between copper and optics is clear to all networkers. One of the practical differences - in an emergency situation, a copper patch cord can be made independently. Yes, we are against samopal, but for hotfix or testing this is done quickly and easily. The optical patch cord is either there or not, so everyone keeps stock in stock.
It’s quite difficult to break an optical patch cord, but once I was lucky: a hand just pulled it out of the connector with meat. Fortunately for me, the network architecture was fault tolerant, the cables went to the server from two shoulders, and the server continued to work. I went to the warehouse and - a surprise! - did not find a single optical patch cord.
He calmly sat down at the computer and drove a patch cord of the required length. And then it turned out that in huge St. Petersburg there is practically nowhere to buy it today.
In the meantime, I noticed a failure on a device that provides server resiliency. One of the ports fell out, and the chess order turned out: one server is unavailable from the first unit, the second server is unavailable from the second.
Cable searches continued at a slower pace. There was only one (Karl!) Place where you can pick up the patch cord TODAY! I had to turn on the RUN AS ADMINISTRATOR mode.
Spare Accessories Rule:
I always keep a supply of optical patch cords. It’s not copper, you won’t be able to quickly squeeze a makeshift.
# 6. Broadcast storm and phones
They launched one piece of iron in the pilot. Where it was spinning, the network was in a single first vlan. Not to say that the monitoring system was absent as a class, there were some rudiments. But it so happened historically that she did not see when the network overwhelmed with Broadcast traffic.
That is exactly what happened. Not without the help of the administrator’s crooked hands, the device gathered a loop on itself, and a Broadcast storm thrashed all weekend.
After that, literally 18 Avaya phones burned out: the CPU melted. Naturally, this is a big hole in the budget. It’s fortunate that the telephone exchange didn’t burn out, - otherwise change the work.
Some users worked at this time. They did not notice the inconvenience, because they worked on old telephones with conventional circuit switching. Well, except that the whole weekend the "grid slowed down" and the reports were sent from the tenth time.
It would seem, what does physics have to do with it? The devices burned physically, because of a physically connected cable in the prod, where playful little hands gathered a loop.
Hardware Setup Rules:
- I keep monitoring always.
- Correctly setting up storm control on BMU traffic.
- If you still have STP *, you need to configure it too.
- Equipment must be managed.
- If experiments on the prod are inevitable, I check what has changed during and after the experiment, and not just go home.
# 7. Search for the lost link
When I worked at a small Internet service provider, there was a box in the attic of one house. The box served as the center of aggregation of all optical compounds. Opening it was dangerous in itself: you can break off the patch cords going to the crosses. But, in addition to everything, this rack was not equipped with organizers. According to the recollections, it was like this:
Attempts to figure out the box worked like an incompletely tested medication: in one place you heal, in another you cripple. It was possible to create an accident out of the blue. Once I wanted to rearrange from port number 2 to number 3, which ultimately led to the failure of the whole region. But that's another story.
Rack & Cross Rules:
- I always use organizers or corner panels. Clearly, it costs money, but then it will be easier to figure out what's what.
- I mark racks, panels, cable, patches, even the entire active.
- I use cable ties and Velcro when laying and securing cables for easy access to equipment and better ventilation in the rack. I use ONLY Velcro for optics.
- I choose the racks for the task, but if the amount of equipment cannot be planned in advance, I take a 42U tall cabinet.
- Mandatory ground in each cabinet and proper installation of shielded twisted pair connectors.
Ideally, I strive for this:
These are not all the stories from the life of the networker. I would be glad if you share yours. What stories helped you not to repeat popular mistakes ?.