I got a lot of these questions on recent IEEE meetup made me decide to write a few so I can cold blooded point you to a post rather risking to forgot something. (Joke of course, these who know me also know I am happy to speak and share experiences till I makes you sleep). To start with a trivia of our consulting company, we had to pick some branding. Internally it was and still is a common joke “Woman for everyone”. Due to lack of sense of humor within IPO office, we had to change that into Infosec with Experience®™

IPO has strict rules...our blog is bit more relaxed

Don’t get me wrong, we still use that joke internally. It comes from the fact most of our customers (specifically ones that establish a first touch) do need help in every possible technology meshup, methodology, appliances, software or programming languages known to earth. Remember you don’t get a green field to deploy things by the book, more frequent than not you face a combination of legacy systems with newer ones, where a lot of guys “tried something”, eventually left your customer’s company. Some of ours are in business for 20 years and more. You get the impression… Domains on same subnets, sync between Windows AD and OpenLDAP, an attempt to make SSO which does not work for say Oracle. One system runs on Oracle, another is on PostgreSQL. Everything is falling apart and they just closed the deal which is about to increase their resources consumption by factor of 10.

You find out core business is within MSSql, with fields that should be Boolean to store enabled/disabled are defined as NVarchar(4000). Full text search for something that should be a bit with binary search of complexity O(log n). Of course, you get a heart attack, followed by successful reanimation, you are back on the keyboard.

Rewriting database and queries need to get through a small door. Modern day developers are used to different frameworks. Nothing against these useful tools to think instead of developer, but over the 15 years, we faced tons of Frameworks backed by strong PR as “best in class” – This is exactly the response you are going to get from developers. Ask me how many “best in the world” frameworks I faced that no-one can’t recall their name anymore? Too many. You can’t blame developers – demand was huge. To school a proper developer who understand what is happening under the hood takes at least 4 years of university followed by years of working in practice. Therefore, get used to. It’s not the code, it’s “some network problem”. They call the function that should do something without understanding what that function doses, if result is not as it should be – blame network. You can’t argue with that, as these who are capable of running tcpdumps, evaluating what’s happening on wire are already long time promoted to more demanding positions.

So, trying to communicate that Java EE is bad, in virtualized environment where you have bare metal, ESXi on top of it, Linux on top of it, a Docker on top (ok this is more encapsulation that virtualization) but… still, and of top of everything Java EE. For a single instruction such as memcpy to get to CPU it’s going to travel round the world and back. Trying to expel a sentence that Java is bad is Heresy. All you are going to end up with is bunch of developers in CEO/CTO office asking why the hell company hired this lunatic. So don’t do it straight. At least not with dev team.

Further, you find a typical “let’s make your job easy” tool such as HQL instead of SQL. What happens when that gets compiled, is another heart attack. Sometimes database will get:

• …where enabled = “Yes” and username = ‘joe’ and company = “somecompany”

some other times,

• …where company = “somecompany” and username = “joe” and enabled = “Yes”

Obviously, this screws up database execution plan, invalidate cache, and instead of hitting the lower dataset and extracting what’s needed, it would do a full text search over a whole table, get enormous set of data – then find what you actually search for. Guess who is going to blame? “Some network issue”, as on their testing table with 500 rows it works instantly.

Reanimated again? Let’s go with multithreading.

This is likely the hardest part you are about to face. Due to all levels of virtualizations, encapsulations and “frameworks”, don’t expect someone will think of what’s happening at poor victimized CPU.

Connect it with previous example. You got a query result with gigs of data. What’s going to be used. Hash map or Hash Reference? Let me tell you – what IDE- of choice offer first. And likely that’s going to be the map. Instead of using a pointer (hash reference) that would hit exactly the memory block you need, a hash will be used triggering a memcpy at kernel level – or if you like tons of gigabytes will get copied between two threads in order to manipulate single record. Who is to blame – anyone. Why, based on assumption since "credible" paid PR article gives an example which shows it's properly used. Don’t expect a Wikipedia link of course. Sharing Wikipedia makes you old man who stuck in 80s.

Let’s get to network stack. Shiny cluster of VMWare looks good right? Everything runs on SAN, you finally think you are on green field. Do the measurements, evaluate io/wait and all these “non-important” things we do. You go through everything with network guys that assures you everything is done by the book. You check SAN, find Jumbo Frames are enabled, check VSphere, jumbo frames are enabled. Looks good right? Well, you login to a switch and find guess what – MTU of 1452. After an hour of explanation, you eventually run:
vmkping -d -s 8972, too long. Lower to 3000 – too long, lower to 1400 works. Upon changes, count of couple of hours explaining administrator their Jumboframes only caused congestion and retransmission.

Managed to get to this point? Now we get to security?

You did subnets, crafted a least access policy, setup Layer 7 firewall, Issued CA on HSM, created intermediate and edge certificates and rolled them as trusted to workstations. We are living in pandemic of ransom attacks and these who do it are smart enough not to use raw connections. Chances are malware is going to get in through encrypted connection – making re-encryption and inspection crucial.

Just give it to a launch break. Someone checked SSL settings and found out certificate is not issued by Google rather an internal edge. This makes you a public enemy number one. This is likely the second time you can find a long line in front of CEO office, waiting to complain against their privacy attack. Explaining that Keys are stored on HSM – that SIEM reacts only in case something gets out of statistical median is not going to help you much. You’ll have to digest it and move forward with what’s best for business. You are not there to be liked – that comes later.

Many developers would find it unreasonably they can’t push the code straight to production anymore. No matter if you are deploying CI/CD, putting statical or dynamical code analysis tools such as SonarQube, you know why you are doing that. Human mistakes happen. Memory leaks too. It’s your task to properly explain both to management and everyone who is facing a change how that change contributes to an overall security.

From Public Enemy to a Hero.

Assuming you did your homework – which gets down to crafting a functional specification, network layout, physical and logical blocks, evaluated, measured and came up with calculation of various parameters over different parts of the system, deployed, patched and fixed the issue – we get to a training, interactive Q/A. In practice we call it “Transfer of information”. Internally, ok I will open up. Transfer Of Information is something we do internally at least weekly while executing the project.

Transfer Of Information - we found crucial to keep internally at least weekly

Transfer of knowledge is where you address all the frustration caused by “changing the bed” for employees. If you did your homework properly during surveillance and planning period you already have at least 5 situations to mention when developer “attacks” why he can’t commit to production rather it has to do via staging environment.

Name them by date when it happened. What was the consequence. How it affected the business. Single comma missing in query ended up with hours trying to find the problem. Have an incident report prepared so you can bring it up, not for a purpose of blaming someone, rather explain rationale as a prevention mechanism.

Be prepared to listen and address every single question ideally by real world scenario that company suffered from. Discuss and ask for inputs. Remember although you have a ton of experience you are not the smartest on earth. Get inputs and discuss as a part of the team. When I say part of the team position yourself in same hierarchy line with others. Remember your call is to help them understand best industry practice not make enemies. Show patience and repeat sessions for as long as it’s needed. You are surely not going to be able to convince everyone what you did is best – no matter how much you point to standards, but you are making a strong ground for what is coming. Remember, they are all aware onboarding a large customer is mission impossible. Different departments have limited visibility and might not be aware of all changes you did. On the other side, you see the whole picture, you have a goal, you have precise measurements and you know what to expect.

A “Hero” to call whenever they are unsure.

The day is come, onboarding of customer begins. No-one believes platform will handle it. At this point even CEO might be skeptical.

Assuming you did your homework, lead through SIT, SAT, FAT, UAT and get them to production. Company had issue handling 10-20 transaction per second facing an outage every now and then. Usually on daily basis. You onboarded customer who is doing 200-300 and everything works like an ice-cold beer.

Support is not messing with 5 customers at the time trying to find an excuse till someone fix an issue. Developers are committing to staging platform where bugs get identified without service interruption. Network guys are not bouncing back with developers who is to blame.

You did it. Skepticism is out of equation. Suddenly everyone seeks your advice. A man with the team who changed things upside down in a month or two and removed all the obstacles that prevented company from scaling. At this point there’s no concurrency with cheaper rates that could replace you. You don’t need to protect by long agreements. Customer is yours to stay.

Now don’t walk one meter above the ground. You never know when they are going to call you with “Hey, we just landed Google as a client”. It’s a whole new level so get back to the drawing board.

Keep in mind. Every problem in computer science can be solved by adding additional layer of abstraction. Except one. - Too many Layers of Abstraction. Most advanced system are actually very simple and modular. You know, these ones you run ./configure and make

Now you scratched the surface. Be ready for everything and do your homework. There's no such things as taking shortcuts. Remember, these guys hired you after dealing with a lot of "smart heads" that tried to bind them to a specific vendor, overcharge, hiding incompetency. Simply, do as you would do if it's your company. If you don't know something, find someone who does. Infosec consultancy is small village. Between all the shiny websites, it's actually the word of mouth that brings business.