Intrusion Detection and Prevention Systems (IDPS): Signature, Anomaly, and Behavioral-Based Detection

Imagine you are the head of security for a very large, very important building. The building has thousands of employees, dozens of floors, and hundreds of rooms full of sensitive documents and valuable equipment. Your job is to make sure no one who should not be there gets inside, and that no one already inside does something harmful. You have three ways to protect the building. First, you have a book of photographs. Every known criminal, every known troublemaker, every face that has caused problems before is in that book. Any guard who spots one of those faces calls the alarm immediately. Second, you have a system that has watched the building for months and learned all the normal patterns. It knows that the cleaning crew arrives at six in the morning, that the finance team works until eight at night, and that the server room is always quiet on weekends. Any break from those patterns triggers an alert. Third, you have a brilliant detective on staff. This detective does not just look at faces and does not just notice odd timing. The detective watches the entire chain of what a person does, step by step, and asks: does this whole sequence of actions tell the story of someone trying to do something harmful? These three approaches are not just a useful story. They are, almost exactly, the three fundamental methods used by Intrusion Detection and Prevention Systems, which security professionals often call IDPS. Understanding how each method works, where it succeeds brilliantly, and where it fails dangerously, is one of the most important lessons you can learn in the field of cybersecurity. Before we go any further, let us be clear about what an IDPS actually is. A computer network, meaning the system of connected computers and devices in a company or organisation, faces threats from the outside world constantly. Attackers try to sneak in, steal data, disrupt services, or take control of systems. An IDPS is a specialised software and hardware system that watches all activity on the network or on individual computers, looking for signs of these attacks. The National Institute of Standards and Technology, an American government body that sets technology guidelines and is commonly known as NIST, describes it as a system that monitors networks and systems for harmful activity and then either reports it or stops it. There is an important distinction inside that definition. An Intrusion Detection System, or IDS, is like a security camera. It sees something suspicious, it records it, and it sends an alert to the security team. It does not act by itself. An Intrusion Prevention System, or IPS, is like a security guard who can physically stop someone. It is placed directly in the path of all network traffic, meaning all the data flowing in and out of the network passes through it, and it can block harmful traffic in real time, in the moment it happens, without waiting for a human to decide. In modern practice, most systems do both things, which is why we use the combined term IDPS. Now, the heart of everything is this: how does an IDPS decide what is harmful? That is exactly where our three methods come in. The first method is called signature-based detection, and it is the oldest of the three. Think back to the photograph book. A signature, in the world of cybersecurity, is a unique pattern that belongs to a known attack or piece of malicious software. Malicious software, often called malware, is software designed to cause harm, like a virus that deletes files or ransomware that locks your data and demands payment. Every piece of known malware has something distinctive about it, just as every person has a distinctive face. It might be a specific string of computer code, a particular sequence of numbers hidden inside the malware's body, or a characteristic fingerprint of the network traffic the malware generates when it tries to communicate with its creators. This fingerprint is called a hash value, which you can think of as a unique identity number calculated from the contents of a file. Change even one tiny piece of the file, and the hash value changes completely. Cybersecurity companies like Cisco, Fortinet, and others employ large teams of researchers whose entire job is to hunt for new malware and attacks in the wild. When they find something new, they study it, extract its signature, add it to a database, and then push that updated database out to all their customers' IDPS systems, sometimes within hours. When your IDPS receives new network traffic or scans a new file, it runs a comparison against every signature in that database. If there is a match, the alarm is raised. If there is no match, the traffic is allowed through. The strength of this approach is remarkable when it works. It is fast, because comparing patterns is computationally cheap, meaning it does not require enormous processing power. It is also highly accurate for known threats, meaning the number of false alarms, called false positives in security language, is very low. A false positive is when a system incorrectly identifies something innocent as a threat. Think of a fire alarm that goes off every time someone makes toast. In security, too many false positives are genuinely dangerous, because they overwhelm security teams with noise and cause them to start ignoring alerts, which is called alert fatigue. But here is the critical weakness of signature-based detection, and it is a serious one. It is completely blind to anything it has never seen before. In security, this is called a zero-day attack. The term comes from the idea that the defenders have had zero days to prepare a response. If an attacker creates a brand new piece of malware that has never been seen by any security researcher anywhere in the world, that malware has no signature in any database. The IDPS will watch it walk right through the front door and do nothing. The WannaCry ransomware attack of 2017 is a powerful illustration of this. WannaCry was a piece of ransomware, meaning software that encrypts, or scrambles, the victim's files and demands payment in cryptocurrency to unscramble them. It used a method of spreading through networks that was, at the time, unknown to most defenders. It exploited a weakness in the Windows operating system, the software that runs most personal computers, and it spread with terrifying speed across the globe. Hospitals in the United Kingdom could not access patient records. Factories in China had to shut down production lines. Shipping companies lost enormous amounts of data. In those first hours, signature-based systems were helpless because no signature existed. But the moment security researchers analysed the attack and extracted its signatures, the game changed. Within hours, IDPS vendors pushed signature updates to their customers, and protected networks could instantly detect and block WannaCry's attempts to spread. This story shows both the power and the limitation of the method in a single dramatic event. There is another weakness worth understanding, called polymorphic malware. The word polymorphic means "many forms." Some malware is written to automatically change its own code structure every time it infects a new machine. It is the same malware doing the same damage, but its internal structure is slightly different each time, like a criminal who changes their appearance slightly before every crime. Because the signature has changed, a simple signature-based IDPS does not recognise it. This forces signature databases to become more sophisticated, using broader, more flexible patterns rather than exact matches, but this in turn increases the risk of false positives. The second method, anomaly-based detection, was designed precisely to address these weaknesses. Instead of asking "does this match a known bad thing," it asks "does this look different from what is normal?" The process begins with what is called a learning phase or baseline period. During this time, the IDPS simply watches. It observes the network and system activity over days or weeks and builds a statistical model of ordinary operations. A statistical model is essentially a mathematical description of what "normal" looks like. The IDPS learns that this server normally receives about ten gigabytes of data per day. It learns that the accounts team accesses the financial database between nine in the morning and six in the evening on weekdays. It learns that the marketing team's computers rarely use certain types of network connections. It builds a rich picture of routine, expected behaviour. Once this baseline is established, the system switches into active monitoring. It now watches in real time and asks, constantly, whether what it sees matches its model of normal. When it finds something that deviates significantly from normal, it raises an alert. The word "deviate" here means to move away from or differ from. Think of a road that curves slightly left over time. Normal traffic follows that curve. If a car suddenly drives straight off the road into a field, that is a deviation. The anomaly detector is built to notice that moment. The power of this approach is immediately clear. Because it is not looking for specific known threats, it can in theory detect anything that behaves unusually. If a completely new, never-before-seen piece of malware starts quietly sending large amounts of data out of the network in the middle of the night, an anomaly-based IDPS can catch that, even with no signature to work from. It can also catch insider threats, which refers to harm caused by people who already have legitimate access to the system, like an employee who decides to steal confidential customer data. An insider threat is particularly difficult to detect by other means because the person has a valid identity and valid access rights. But if their behaviour suddenly changes, if they start accessing files they have never touched before, or downloading unusual amounts of data, the anomaly detector will notice. A clear real-world illustration comes from the scenario of data exfiltration, which means the unauthorised copying and sending out of data. Imagine a company's main database server has a firmly established normal pattern of sending out about ten gigabytes of data each day. One night, the anomaly-based IDPS detects an outbound transfer of five hundred gigabytes of data flowing to an unknown external IP address. An IP address is a number that identifies a specific location on the internet, like a postal address for a device. This five hundred gigabyte transfer is so far outside the normal range that the IDPS generates a high-priority alert immediately. Security analysts investigate, discover the transfer in progress, and interrupt what would have been a catastrophic data breach. Without anomaly detection, this attack might have succeeded silently. Yet anomaly-based detection carries a serious problem of its own, and it is the same problem that has frustrated security professionals for decades: the rate of false positives is high. Very high. Think of the burnt toast problem. A smoke alarm cannot tell the difference between a kitchen fire and overdone bread. It simply detects smoke above a certain level and screams. An anomaly-based IDPS is similar. A system administrator, meaning the technical person who manages and maintains a company's computer systems, might choose to perform a large backup of all company files late on a Sunday night to avoid disrupting business hours. This is a perfectly legitimate action. But to the anomaly detector, it looks exactly like data exfiltration. Enormous outbound data transfer. Middle of the night. Sudden and unusual. Alert triggered. The security team wakes up, investigates, and finds nothing wrong. Now multiply that by dozens of similar events over a month. The security team starts to tune out the alerts. And then one night, the real data theft happens, and nobody pays attention because they have been burned by false alarms too many times before. There is also a subtle and clever attack technique that exploits the anomaly detection method directly. It is called a slow and low attack, sometimes called island hopping or base-rate manipulation. A sophisticated attacker who understands that their target uses anomaly-based detection can choose to conduct their attack incredibly slowly. Instead of stealing five hundred gigabytes in one night, they steal half a gigabyte per day over many months. This is below the threshold the detector uses to define "unusual." Even more cunningly, they do this consistently enough that the system eventually incorporates this slightly elevated data transfer into its model of "normal." The attacker has, in essence, trained the security system to accept their theft as routine. This is a sobering limitation, and it is part of why the third method became necessary. Behavioural-based detection is the most sophisticated of the three methods, and it represents where modern cybersecurity is actively evolving. It is often powered by machine learning, which is a branch of artificial intelligence where computer systems learn patterns from large amounts of data rather than following simple pre-written rules, and artificial intelligence more broadly. The key difference between this method and anomaly detection is context and sequence. Anomaly detection asks whether any single action is unusual. Behavioural detection asks whether a whole chain of actions, looked at together and in order, tells the story of an attack. To understand this, think of how a detective works compared to a simple alarm system. A smoke alarm does not care who set off the smoke or why. It just responds to smoke. But a detective would notice that first a specific person entered a building, then that person went to an unusual part of the building, then they attempted to open a locked door, then they handled objects they should not have had access to, and then they tried to leave through an unusual exit. No single one of those actions, in isolation, might be enough to arrest someone. But the sequence, the chain, the story that those actions tell together, is highly suspicious. That is exactly what behavioural-based detection does. This method is often guided by something called the MITRE ATT&CK framework. MITRE is an American non-profit organisation that works on technology and security issues. The ATT&CK framework, where ATT&CK stands for Adversarial Tactics, Techniques and Common Knowledge, is a large, publicly available library that documents the specific methods that real attackers use when they are trying to break into and operate inside a target network. It describes, for example, how attackers typically gain their first foothold, how they move from one machine to another inside a network, a process called lateral movement, how they try to gain higher levels of control over systems, which is called privilege escalation, and how they eventually get data out. Behavioural detection systems use this kind of structured knowledge to recognise attack campaigns in progress, even when each individual action looks innocent. Here is a concrete example of how this works. A behavioural-based IDPS might observe the following sequence of events. First, a user account logs in from a country that this user has never accessed the system from before. By itself, this could simply mean the employee is travelling. Second, within minutes, that same account attempts to access a sensitive file storage area that it has no history of ever touching. This is slightly unusual, but perhaps the employee has a new project. Third, the account then runs a PowerShell script. PowerShell is a powerful command-line tool built into Windows that system administrators use for automation, but that attackers also love to use because it can do so much so easily. A first-time user running a PowerShell script is more suspicious. Fourth, the script attempts to create what is called a back door, meaning a hidden method of accessing the system later even if the current access is cut off. Fifth, the account attempts to establish an outbound connection to an IP address that is listed in threat intelligence databases as belonging to a known attacker's infrastructure. Now, look at that chain again. Unusual login location. Access to sensitive files. PowerShell execution. Backdoor creation attempt. Connection to a known attacker server. Each step adds evidence. Together, they form an unmistakable picture of an attacker who has stolen or guessed a user's login credentials and is now working their way through the network. A behavioural system can recognise this pattern and raise a high-confidence alert, or in an IPS configuration, can block the account and the connections automatically before serious damage is done. This kind of threat, a long, careful, multi-stage attack conducted by skilled attackers who are patient and deliberate, is called an Advanced Persistent Threat, or APT. The word "persistent" is important here: these attackers do not try to smash through the door in one moment. They get in quietly, stay hidden, and work slowly toward their goal over days, weeks, or even months. Nation-states, meaning governments conducting espionage or sabotage, and highly sophisticated criminal organisations are the typical sources of APT attacks. They are the hardest of all threats to detect precisely because no single action they take is necessarily alarming in isolation. Behavioural detection systems designed for this kind of deep, endpoint-level analysis are often called Endpoint Detection and Response systems, or EDR, or their expanded cousin Extended Detection and Response, known as XDR, which also pulls in data from network monitoring and cloud services. These tools give security analysts something like a film of the attack rather than just a photograph. They can see the whole story of how an attacker moved through the organisation, which machines they touched, which files they accessed, and what they were trying to achieve. This makes the security team far better equipped to understand the full scope of the problem and close every hole the attacker used. The major cost of behavioural detection is complexity and computing power. Watching every action on every machine, correlating those actions across thousands of devices, and running machine learning algorithms on all of it simultaneously requires enormous resources. It is expensive to implement and requires highly skilled people to manage and interpret the results. There is also a pointed debate in the security community about whether artificial intelligence is really the breakthrough it is sometimes claimed to be. Some professionals warn against treating AI as a silver bullet, meaning a magical solution that solves everything. Attackers are increasingly studying how AI-based detection systems work and designing what are called adversarial techniques, meaning methods specifically designed to fool the machine learning model. Just as a person might learn to walk calmly and act innocent to fool a human security guard, attackers can craft malicious traffic that is carefully designed to look normal to an AI system. The arms race between attackers and defenders never ends. A further debate running through all three methods is the question of privacy. Anomaly and behavioural detection, by their very nature, require watching everything. Every login, every file access, every message, every connection is potentially a data point that feeds the system. In countries and regions with strong privacy laws, like the European Union's General Data Protection Regulation, known as GDPR, this creates genuine legal and ethical tension. Employees and citizens have a right to privacy. Organisations have a duty to protect their data. Drawing the line between necessary security monitoring and unacceptable surveillance is not a technical question. It is a human one, and it remains actively contested by lawyers, ethicists, and security professionals alike. The honest conclusion of all this is that no single method is sufficient on its own. The modern cybersecurity field has largely arrived at what is called a defence in depth approach. The phrase means layering multiple different types of defences so that if one fails, others are still in place. A typical well-protected organisation might use a signature-based IDPS at the edge of its network, the border where the internal network meets the wider internet, to catch the vast majority of known threats quickly and cheaply. Behind that, it might deploy anomaly-based monitoring to catch unusual patterns in traffic volume and timing. And on each individual machine or endpoint, it might run a behavioural EDR tool that watches what users and processes actually do and correlates those actions into meaningful stories. Together, these layers provide far stronger protection than any single method alone. Think of it like the security system for that building we imagined at the start. The photograph book of known criminals is still useful and catches many threats immediately. The guard who knows everyone's routine adds another layer. And the brilliant detective who watches for the chain of suspicious actions provides the deepest, most sophisticated protection of all. Remove any one of those layers, and the building is more vulnerable. Keep all three, and the attacker's job becomes extraordinarily difficult. To bring all of this together into a clear summary of the key ideas: an IDPS is a system that monitors networks and computers for harmful activity, either alerting defenders or actively blocking threats. It can work in three main ways. Signature-based detection matches activity against a database of known attack patterns, and it is fast and accurate for known threats but completely blind to new ones. Anomaly-based detection builds a model of normal behaviour and alerts when something deviates from it, which allows it to catch unknown attacks but generates many false alarms. Behavioural-based detection looks at sequences and chains of actions to understand likely intent, and it is the most powerful method for detecting sophisticated, multi-stage attacks, but it is also the most expensive and complex. Real-world attacks like WannaCry show how signature detection can be both a lifesaver and a limitation depending on timing. Events like large-scale data theft illustrate where anomaly detection proves its value. And the threat of patient, skilled nation-state actors demonstrates why behavioural detection has become essential. The wisest approach is to layer all three, understanding both what each one can do and, crucially, what each one cannot.