In other news: If I can intercept the log messages (unencrypted remote syslog, or log file public accessible, log messages also shown to the user) I can still just inject stuff like "${env:DATABASE_PASSWORD:-xxx}". Lots of people are only focusing on the aspect how this can be abused to inject arbitrary code, while it is actually a similar class like SQL injections: Blindly trusting user input instead of doing proper sanitization.
(The whole discussion why a logging mechanism should support the whole JDNI stuff is still perfectly valid).
Please stop suggesting that SQL Injections are a sanitization problem. They are a problem of escaping in the application constructing the query, not a problem with whatever the user has typed in.
The log4j vulnerability shares some common traits, but it's completely different at the core. It's not a problem of sanitization, and I'd argue it's not a problem of escaping either (though escaping could fix it). It's a problem with an obscure feature that is left with insecure defaults, and is (was?) unknown to the vast majority of developers who integrate the library in their apps.
Part of the problem is (AIUI) that expansion of ${} type constructs is carried out after parameters are inserted into the log message's format string. I don't believe the use case requires that, i think it was just easier to implement. If that expansion was done on the format string, before parameters were inserted, this attack would not be possible. Injecting dodgy strings into log messages as archi42 suggests would also not be a problem if the insertion was done safely.
Agreed that this is not a sanitisation attack, but i think that means it is a format string attack:
Hah, linking to OWASP is a good point. I have now put some more thought into "how would I EXACTLY classify this, if I had to list it as a finding for a customer".
Now, I think format string is not entirely right, either. That's more related to `printf`, which splits code (the format string) and data (the varargs). But log4j actually mixes code and data into one string. In the OWASP frame work, I think the more generic variant of format string attacks would be https://owasp.org/www-community/attacks/Code_Injection (that's actually linked under "related" for the format string attacks).
OWASP lives mostly in the web world, but links to CWE-77 (Command Injection, https://cwe.mitre.org/data/definitions/77.html), which is pretty generic. And log4shell matches the description just nice: 1. Data from untrusted source: yes; 2. data is part of a string that's executed: interpreted, which I think qualifies as a yes; 3. the execution gives capabilities the attacker would not have other wise: oh yes!
So it's probably safe to claim that this could fall under CWE-77 (too bad CVEs rarely use CWEs).
Now, CWE-77 (Command Injection) is a child of "CWE-74: Improper Neutralization of Special Elements in Output Used by a Downstream Component ('Injection')" https://cwe.mitre.org/data/definitions/74.html. Which states "The most classic instantiations of this category of weakness are SQL injection and format string vulnerabilities." - that's probably why the both of us thought of the the two of these! :) And even if CWE-77 is to specialized (this could be argued) CWE-74 should be a good match, quoting CWE-74: "The software constructs all or part of a command, data structure, or record using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify how it is parsed or interpreted when it is sent to a downstream component" [end quote].
Funnily, CWE-74 is a child of https://cwe.mitre.org/data/definitions/707.html - "Improper Neutralization". Which says neutralization can be done by (among others): "[...] transformation of the input/output to be "safe" using techniques such as filtering, encoding/decoding, escaping/unescaping, quoting/unquoting, or canonicalization [...]"
I disagree with you: IMHO "escaping user input in the application constructing the query" is one possible form of sanitization. Why do you think it is not?
Also I said "similar class", not "strictly equivalent". Both instances are a case of "user input handled as code, not as data". Now with SQLi the problem is widely known (but sadly still happens) and we have techniques like parameter binding to outright avoid the problem. The same techniques are among the <edit>theoretical</edit> possibilities to avoid the problem with log4j.
Though I agree, the problem are the insecure defaults (JNDI enabled) and that seemingly every java dev used log4j but wasn't aware of the "${}"-expressions. I just pointed out that this "trend of neglect" [sry, non-native speaker here<edit>, but I hope you get what I mean?</edit>] seems to continue: Everyone and their CISO is now aware that attackers can abuse JNDI and the whole mitigation recommendations (<edit>e.g.</edit> remove JDNI classes) so far was primarily focused on that. Still, even if JNDI wasn't a thing, the "${}"-expressions can still be abused (it's just more specialized now and not a huge pwn fest), requiring mitigations beyond disabling JNDI. Just look at the reporting for more examples of ignorance: When this started to explode, there were some who thought blocking the string "${jdni" in their WAF would be enough (and maybe it helped against script kiddies), but a smart attacker just used/uses a string like "${${env:randomstring:-j}dni[...]" defeating the simple filters...
Yeah, people should stop talking about sanitation in relation to SQL, because that is how we end up with data like "OConnor". The two correct solutions are escaping and parametrization/prepared statements (i.e. sending the parameters out of band).
Huh? I understand "sanitation" to mean "a transformation of data that makes the data safe for use in the subsequent program". Escaping is one way of doing that transformation, and it's good because it's not lossy.
Another example: A "transformation" that's a "sanitation" but not "escaping" would be replacing all occurrences of "<" with "<" (among others!). It surely doesn't add escape characters (e.g. \), but instead replaces the problematic substring with a replacement string that makes the string safe to display on a website. Of course you'll want to replace user-supplied "<" with "&lt;".
(btw, thanks for that it's "sanitation" and not "sanitization" ^^).
Not sure I agree, for me sanitation has a strong connotation with removing things. And sending data out of band from code on the other hand cannot be seen as a form of sanitation. And is my preferred method of solving this issue.
You're essentially saying "sanitation" equals "filtering". Looking at CWE-707 (https://cwe.mitre.org/data/definitions/707.html) I'd rather say that "sanitation" is what MITRE calls [begin quote]transformation of the input/output to be "safe" using techniques such as filtering, encoding/decoding, escaping/unescaping, quoting/unquoting, or canonicalization[end quote] (well, I'm repeating myself here).
But there are also results where it isn't really clear, or where the only sanitation technique considered is filtering. So I'd say "yeah, it's unclear and poorly defined".
Buuuuut: I still like my definition more, as I have a word for "all techniques that aim to make an input safe for processing" (sanitation/sanitization) while I can still refer to "destructive elimination of substrings" as just "filtering", which is a again different from outright "rejection of input" by using an "allow list" or "deny list". :P
I agree that splitting data and code is the way to go, if that's an option. But I didn't talk about that in the post you're answering to, so I'll ignore that ;-)
I think, generally, sanitization means protecting against potentially malicious input. Whether it takes form via escaping or removal or some other remediation is beside the point.
I'd say sanitization implies that there's something wrong with the data and you need to clean/fix it. But the former isn't true and the latter sounds like removing or irreversibly remapping characters.
> Please stop suggesting that SQL Injections are a sanitization problem. They are a problem of escaping in the application constructing the query, not a problem with whatever the user has typed in.
That applies to the <2.15 mitigation for disabling message lookups. That mitigation for message lookups does not affect context lookups. Nor does the 2.15 fix for disabling, nor the 2.16 fix for removing message lookups.
But 2.16 disables JNDI lookups entirely, so that they cannot be triggered via any lookups, including context lookups. But context lookups can still trigger other non-JNDI lookups.
People think they are safe because they don't allow for outgoing ldap, but combined with a way to do dns lookups, these kind of things can get really scary.
The java ecosystem is full of these abstractions, that are not outright security problems in themselves, but give rise to complex interactions that makes it hard to judge what the implications are.
This particular logging system has been used by millions of developers for the better part of a decade before any one single person realized the actual implications.
What i discovered was all of my applications that use log4j do not use version 2. They have out of date dependencies or use slf4j. I'm not sure how i feel but mostly happy im not scrambling. I wonder what adoption rate v2 actually has.
It scares away people who see 'GPL' and refuse to read further, perhaps. Not only is it dual licensed, but both licenses (EPL and LGPL) freely allow linking with proprietary software.
Maybe - but if you don't know what QWER is and I explain that "QWER is just like ASDF" but you don't know what ASDF is either, I haven't really clarified anything for you.
This is somewhat intended, though. Also, for intercepting the logs you need _a lot_ more access than for an unauthenticated RCE and that database might not actually be accessible from the outside.
It's not cool, but far less severe than the original issue.
Yes, absolutely. log4shell is a pwn fest that makes the recent Exchange debacle look rather innocent (and that was already quite bad). But I can envision scenarios like unencrypted rsyslog on a LAN being relevant during pentests, or misconfigured (read: attacker readable) log files.
Are you saying that log4j is also used for analyzing logs that have already been generated ? My understanding was that it was intended for use inside an application for generating log messages and processing them before they are sent out to a file or syslog.
Logstash is a hugely popular application for collecting/processing/shipping logs, mainly associated with elasticsearch ("ELK") but has lots of output plugins. It uses log4j, so if one of the log events it was processing contained an exploit string, and also caused logstash to encounter an error processing or shipping the message*, that message would get logged to logstash's own log, thus triggering the exploit deep in someone's infrastructure.
I had the same thought. However, I think it's a more generic attack. If you can persist or inject the jdni strings, there's a chance they will be emitted by another component which analyzes the logs and has a log4j dependency elsewhere in the stack.
I've heard people talk about this problem with analyzing windows event logs using Java-based software.
That's interesting. It seems a bit weird that a log analyzer would need to do additional logging, but maybe that's the case in larger systems. I wonder if using log4j as a tool for log analysis as opposed to generation might be a shadow use case of the library. You could have an app that parses log files, converts them to Event objects and then processes that event stream through log4j. I have no idea whether that would be terribly useful and is actually being done by anyone but I guess that's one scenario where these CVE's could cause delayed effects.
This vulnerability will likely turn out to follow the trajectory of the Meltdown and Spectre vulnerabilities from...(checks google)... 2018.
I speculate will be a cluster of newly discovered vulnerabilities that follow a similar pattern, possibility across a number of familiar Java libraries.
What I have been pondering again in the past few days is dependency management. For many projects this is a open pom file pop in the new version, recompile, deploy and bobs your uncle you are good.
But some projects are little more interesting in that they include libs like this, or include libs that include it. It is part of their foundation/framework. So you can still put in override and ignore what the lib wants.
But now when it comes time to upgrade that top lib maybe there are 6 newer versions of the one problematic lib you were concerned about today, but now it is a year later and you forgot about it. What is the mental context to say 'oh yeah go back and check that'. You may have got used to ignoring the 'hey you are overriding that jar file'. Are you going to do that for all of the libs/jars this top level framework drags in?
It is not just java that has this sort of issue. NPM (though you can float the versions at least), nuget, rust, C++ containers, etc. A lot of projects out there have taken on this maven central style building. Which is a huge productivity boon. But in many ways has not helped us with dependency hell, and in some cases has made it wildly worse. I see simple projects pulling in 150+ items in some cases. What is the mental context of actually managing that? And have real fun if your project depends on a 'dead' project where there are little to no updates.
This one is at least high profile enough that many people are revisiting old build chains that have long ago been forgotten and updating them. That will kill out many other vulins that have been lurking. But many will just do that one jar set and call it a day. But you are probably right we are going to get see a few of these style of attacks.
There isn't one. This method of development inherently leads to these kinds of problems. Software engineering is still in its infancy and has yet to truly develop a universal best practice development cycle. Eventually these things will be codified.
Is there actually evidence for this? The 30 year arc of the industry to date that I am familiar with so far shows little sign of engineering standards in software codifying around anything.
If we are in the infancy stage in 2020, we are one very large infant. More likely to me is that in the future, software will change so much again that what we do today is unrecognisable; not because of codification but because of the inevitable tech churn between now and then.
The closest things we get to standards are automated vulnerability checkers. I'm currently dealing with one of these vulnerability scanners which claims our code base has some XSS vulnerabilities, which I know is wrong because the app isn't a web app. SMH
Regulations are written in blood. Eventually enough people will die from bad software that it will be regulated like engineering. The more we depend on computers the more likely this is to happen.
Mmh, I don't think so. Beside logging most other libraries will already sanitize user input since it is a more commonly known attack vector for those kind of libraries. I would compare the vulnerability to https://github.com/frohoff/ysoserial.
What they have in common with spectre is that: Suddenly it was realized that the design was all wrong. What was thought of as a feature, was a giant vulnerability.
However, log4j seems to be adjusting their design rather quickly, so they should be out of it rather fast.
The recent log4j vulnerability really piqued my interest, and I've spent the last few evenings working on a proof of concept Java agent that could mitigate similar vulnerabilities in the future, for applications that are able to completely forego platform features like JNDI, serialization or native process execution.
It's not a lot of code, but it uses parts of the platform that I think are a bit unusual for most devs, so it was quite interesting to implement. Happy to discuss details, ideas, and concerns.
One idea for a possible improvement is to make the feature block list adaptive, i.e. watch what the application uses in the first few minutes of execution, and then shut down all unused "dangerous" features for the remaining lifetime of the VM. Not sure how reliable this would be though, especially for services which have background jobs that might only run once a day.
If you mean it's turning into a grinding and repetitive groundhog day, I concur.
We don't even use log4j and yet are fielding a steady stream of requests to reassure clients that we've mitigated against it, which hasn't been an issue because we don't use it (although we do use log4net with Stackify and it's certainly a possibility that may be compromised in future), and of course are having to audit all our own suppliers.
I encourage companies to post information on their websites about this so we don't all have to keep emailing you (I'm trying to get some info up on ours detailing mitigations, for similar reasons; once it's there I can send some all company comms and hopefully never see another email in my inbox with "log4j" in the subject line).
Of course, this whole issue is crucially important to deal with, but it's also frustrating because it's an entirely non-value-adding time-vampire.
More realistically, this code is 20 years old, several generations of maintainers worked on it, accumulating patches over patches over patches and nobody wanted to be the guy that say No / we should remove that feature.
You can see the discussions on HN about Gnome removing features as a good example about how hard it is.
Actually, it’s not. Log4j 2 is a ground up rewrite of Log4j and is less than 9 years old from when the rewrite started, 7 years for the GA release. Log4j 1.x isn’t vulnerable as it doesn’t have this “feature”.
Imho, this isn’t a “patches over patches” situation, it’s a very poorly thought out “feature” that almost nobody needed and shouldn’t have been added in the rewrite. The concept of “separation of concerns” exists for a reason. Your logging library should _just_ be concerned with logging.
>You can see the discussions on HN about Gnome removing features as a good example about how hard it is.
The issue with GNOME is the insincerity of feature removal, things like "nobody uses that", "nobody needs that", "you are doing it wrong" instead of just be sincere , "file picker is a usefull feature but we can't or don't care to implement it" , "optional server side decoration is a good feature but our devs want to work on cool shit not backwards compatibility" ... if you are an open source dev, be sincere and people will not be disgusted by your community and you don't get the go to example for bad developers/community.
I'm not familiar with the GNOME story, but removing features takes courage and is actually a form of sincerity.
A developer team can be insincere and avoid outrage by just leaving half-broken features in the software, stop testing them, and ignore all complaints about it. Low quality and inactivity doesn't make for as good a story on social media.
>but removing features takes courage and is actually a form of sincerity
That is false, say I remove the feature X because reason A but I tell everyone that is for reason B. So I had the courage to remove X but I did not the courage to be sincere, maybe I am not sincere with myself and the real reason A is that the code was written by somebody else in the past and I want to implement new cool shit and not read somebody else code, but I can't admit that I don't care about feedback because I might be replaced as a project leader so I invent bullshit excuses like blame the previous developers that wrote bad code, blame the user that are using it wrong or invent some bullshit UX story with no real user tests to push my egotistic vision on everyone else because I am really an Apple fan but I could only get a job to work for RedHat
This JNDI "feature" is 8 years old. It was merged in 2013 from what I have seen, and has been defended by some as an "useful" feature. It has been known since at least BlackHat 2016 presentation that this is an incredibly dangerous and stupid thing to keep.
cricket noises nobody reacted until now when minecraft servers got hacked.
> and has been defended by some as an "useful" feature.
I would say the feature is useful but the implementation is insane. I would expect a SQL prepared statement approach, only contents of the statement / format string are resolved. Interpreting variables in user provided input just asks for SQL injection attacks and needs to be either prohibited or extremely restricted.
I guess you are referring to this[0]. I am surprised if it was known that early then why did it take so long to find the issue? Or were exploits already run earlier and nobody reported?
When I was shopping for groceries today, I saw something strange. The buses have displays to show what Line they are, today all of the buses I saw had blank displays. On my way home, I could see the display on the bus driver's seat, which only displayed an otherwise blank Windows Embedded CE 6.0 start screen.
Oh, and the thingy you put your ticket in get the stamp was out of order on all the buses.
Made me wonder if that was due to some component involved was using log4j? I don't think it's very likely - why would the bus display or the ticket stamping machine be affected in a way that requires turning them off? But who knows, maybe somebody overreacted, or maybe these depend on some backoffice infrastructure that is temporarily offline due to log4j.
It was weird, though. This is the first time I remember a security vulnerability made the national news (in Germany, if it matters). I don't think even Heartbleed got that much attention outside IT news outlets.
I must confess to feeling that the reach of this CVE is a bit oversold. Yes, it's a terrible RCE and a major catastrophe for anyone using log4j. But log4j hasn't been the popular choice for logging implementations in a long time. Libraries typically use facades like slf4j or commons-logging, so the risk of pulling in an active log4j implementation transitively is pretty small. Your application/appserver either logs with log4j or not, and your developers are probably aware of this when they configured logging. You shouldn't need to pour over code to figure it out.
It would be far worse if this exploit existed in slf4j or commons-logging.
The major risk here probably isn't in-house developed code but versions of log4j used in a vast array of packaged software and systems.
https://github.com/cisagov/log4j-affected-db <-- that's a (probably incomplete) list of possibly affected vendors and applications, and in many cases the vendor can't even say yet whether they're vulnerable.
Yup. Here's a concrete example -- my team owns a couple of java services. They're spring boot with slf4j as our logging wrapper, using the default logging implementation of logback -- we didn't opt for log4j. So we're good, right? Well, let's do a `gradle dependencies` and see if we're pulling it in transitively -- still good. See some `log4j-api`, but no `log4j-core`. We're not actually doing any logging with log4j -- the "api" is from spring to allow you to use log4j if you want. Again, we're not, so we're fine.
Oh, but it does turn out that we do have the elastic apm agent, and it has log4j as a shaded dependency. So not as free and clear as we thought, and damn hard to detect that.
Shaded dependencies are a nightmare, and it's a particular bone I have to pick with using Elastic products that they think this practice is ok.
I prefer to package all my projects into a single fat jar, since it makes distribution so easy and clean, but if I was creating jar files with the expectation that other projects would use them I would make sure to package everything separately.
In this case using fat jars made things really easy to check for the effected classes and luckily for me I also didn't have any log4j-core classes in my fat jars.
log4j-api isn't actually a problem (directly). It's the usage of log4j-core that has the JndiManager class that's the injection point for RCE.
In the case of the transient dependency, it's further complicated because we've seen that some packages repackage the JARs they depend on... that means that statically analyzing for log4j is very difficult because you can't use hashes (even if you unzip the jar and hash class files directly).
I've been working on a scanner for this stuff on GitHub[0], and it's a real pain in the neck lol. Especially for Vendor software that you don't control.
log4j is in widespread use, directly or indirectly.
Most people don't give a crap about the exact Java logging library, the only places that actually care are those that bike shed, since functionally Java logging stuff is generally quite robust and full featured.
The worst part about my previous phrase is that the places that don't care about the Java logging library frequently don't do security audits, so prepare for a ton of exploits and data leaks in the following months and years.
Using logging facades means that libraries don't need to update -- which is great -- but libraries were never directly vulnerable anyway. log4j is, to my knowledge, still by far the most common actual implementation of logging in the Java ecosystem. The assertion that it's not popular for logging only holds if you assume that logging facades are logging implementations, which they are not.
My completely-unverified guess would be that there are more people immune to this issue because they never migrated past log4j 1.x than there are who are immune because they picked up Logback or something similar.
> You shouldn't need to pour over code to figure it out.
This is true but as sibling comments have pointed out, a lot of other software you might be deploying without having written or configured the logging for are written in Java.
Most people using e.g. spring boot or quarkus would end up using the defaults that come with those frameworks. For spring boot, the default is actually logback. However, you can switch it to log4j2. https://spring.io/blog/2021/12/10/log4j2-vulnerability-and-s...
Log4j2 never quite got the same status that v1 had. V1 should be considered a bit obsolete at this point. It still works of course but it has some performance issues that both log4j2 and logback try to address.
The issue with high profile vulnerabilities like this is that there are a lot of projects where dependencies are rarely updated.
I update aggressively on my own projects to stay on top of changes and keep the effort related to mitigating compatibility issues at a minimum. A nice side effect is that you get all the latest security, performance, and other fixes. In my experience, updates get harder the further you fall behind. So, the longer you wait, the more likely you will have a lot of fallout from updates and the more likely it is that you will be exposed to pretty serious bugs that have since been addressed upstream.
If you are like me, I can recommend the excellent refreshVersions plugin for gradle. It makes staying on top of dependency updates a breeze. I run it every few weeks to spend a few minutes updating misc libraries, and verifying everything still works. Run the command, update to the suggested versions, create a pull request and merge when it works.
Occasionally there are issues with specific libraries but 95% of the updates are completely painless and the remainder are usually pretty easy to deal with. And if there are show stopper issues, I want to know about them and document them why we can't update.
I would recommend doing the same for packaged software. I work with a lot of customers running ancient versions of whatever for no other reason than that they seem a combination of fearful, ignorant, and indifferent about what will break because they can't be bothered to even try. Mostly updating them to more recent versions isn't that big of a deal and it tends to address a multitude of performance, security, and other issues.
> For spring boot, the default is actually logback.
I did not know this, thanks for letting me know!
> I update aggressively on my own projects to stay on top of changes and keep the effort related to mitigating compatibility issues at a minimum. A nice side effect is that you get all the latest security, performance, and other fixes. In my experience, updates get harder the further you fall behind.
This is absolutely a best practice, though I think people struggle with it for all sorts of reasons. In general one of the downsides of maintaining a diverse codebase is that this constant update cycle becomes more and more difficult, and it's one of the things that I find drives towards more consistent tooling within a team.
> I work with a lot of customers running ancient versions of whatever for no other reason than that they seem a combination of fearful, ignorant, and indifferent about what will break because they can't be bothered to even try.
While I agree this is something people need to get over, we have to take some blame for this as an industry. A lot of people have bad experiences with upstream Shiny Object Syndrome.
That's my observation. My favourites are the projects where the last commit was 5 years ago with a copy-pasted log4j stanza that must have been 4 years old at that point, which now can't practically be upgraded at all because of the bitrot and loss of organisational knowledge. I've seen one that needed somewhat special measures just to regain access to the source code...
I can't say I'm feeling the same. Still lots of people testing over at https://log4shell.tools almost a week after this vulnerability became widely known. Plenty of people still discovering they're vulnerable as well. I think it's likely that these are just the people who know they're using log4j. If you're running a black box product from some vendor you'll have no clue you're vulnerable until it's too late.
I have ported a Spring Boot application from log4j to logback and it was pretty easy since we used slf4j. The code was the same, the only thing I had to do is to convert the logging configuration. The features mapped one to one and it to some fidgeting to make it work, but all in all it was pretty straight forward. Would recommend.
The fact that putting these strings in the name of an iPhone or a Tesla was vulnerable suggests to me that there are plenty of vulnerable systems out there.
My initial reaction was also tentative skepticism. Not that the package isn't popular, but that it must be hard to find real world interactions where you can successfully get a payload in, and a lot fewer still where you can actually execute it.
Yet we have seen a large number of high profile hacks already, and they keep coming. Turns out that not only are there many ancient java versions in use, the aren't very compartmentalized either.
Is there a list of successful hacks anywhere? Most of what i've seen on Twitter is someone posting an attack string to a website, then showing a record of a DNS lookup, which is some distance from actually being a successful exploit.
(The whole discussion why a logging mechanism should support the whole JDNI stuff is still perfectly valid).