Event Espresso and WordPress 5.0 Development

For anyone developing in the WordPress space, you should be well aware by now that WordPress 5.0 is bringing with it some pretty significant changes. Here at Event Espresso, we’ve been working hard to keep up with those changes and prepare our products for the future brought with those changes. In this post, I want to give a bit of an overview of what those changes mean for Event Espresso and how that might affect you as a developer working with our products.

In the short term, it’s important to highlight that nothing specific needs to be done for any Event Espresso customizations:

  • Currently, our editors are unaffected by the new WordPress editor. So any customizations in them (meta boxes, etc) will continue to work after the launch of WordPress 5.0
  • All our shortcodes continue to work in both the classic and new editor. The new editor has a shortcode block that can be used with shortcodes. One distinction, however, is that shortcodes exist on their own block. In most cases, that shouldn’t affect Event Espresso shortcode users too much because EE shortcodes tended to be their own distinct content pieces anyways.

So, in the short term at least, you can have a moment of relief! However, we do think there are some things you should be planning for and accounting for in the future.

Javascript and React

The new WordPress editor is javascript heavy. It goes without saying that unless you are familiar with javascript, you’re not going to have an easy time developing for the new editor. In keeping with the shift in WordPress towards more javascript heavy functionality, we have been leveling up our skills significantly in javascript and in particular, the new “blessed” javascript framework within the WordPress environment which is React. While you don’t need to be familiar with React in the post WordPress 5.0 world, understanding it greatly helps with developing complex things in the new editor and is definitely needed if you want to contribute to either the editor itself or things built for it. I’ve been fortunate to be able to contribute in different ways to the Gutenberg editor and can definitely say knowing react has made that easier.

What does this mean for Event Espresso?

  • We’ve switched our javascript development to a more modern process. This is surfaced in the implementation of a Webpack build process along with a more modular design.
  • For all new javascript in Event Espresso and our add-ons, we will be using react and/or the WordPress packages that help accomplish our goals.
  • Existing javascript in use in our products will still be maintained, but we will favor refactoring to newer javascript where and when we are able.

Extensibility (including shortcodes vs blocks )

As much as possible we are going to try to maintain parity with our existing extensible functionality in our products. However, there will be places where it is simply not possible to provide the same functionality in the move to more javascript heavy functionality. When that happens, we will communicate on this blog those cases.

An example where this will be happening in the short term is the introduction of Blocks in the new editor (vs. WordPress shortcodes). One of the significant features of the new WordPress editor is the introduction of the concept of Blocks. We’re “all-in” with this new feature which means:

  • We will no longer be adding new features to any of our current shortcodes. We’ll still do any bug fixes that may be needed, but none of our shortcodes (or their related templates) will have additional feature development.
  • We will keep existing shortcodes in our products for as long as WordPress supports them.
  • We are not introducing 1:1 parity with our existing shortcodes when introducing new blocks with similar functionality. Blocks (such as the Event Attendees block) will exist entirely on their own and the layout, styling etc is not inherited from the equivalent shortcode.

Going forward, we’re iterating on and developing blocks for the new editor because we believe the user interface and experience is much greater than that of the WordPress shortcodes.

Practically, this means for you, the WordPress developer:

  • If you are already using the extensibility in existing templates for the current shortcodes, you can continue to do so going forward (especially since the new editor does support WordPress shortcodes).
  • Initially, the extensibility on the new blocks will be limited (styles will be controlled via CSS, etc.) until we get a handle on what kind of extensibility is needed by our developer users (which includes ourselves!). So we do plan on having extensible blocks, but the shape of that is still yet to be determined. What we can tell you, is that its very likely all extensibility for our blocks will be javascript only. In other words, it’s unlikely templates will be modifiable PHP side (we’re open to feedback on that over time).
  • With that said, we do have a PHP framework built for more easily registering new blocks PHP side (more on that in the documentation section).

Documentation

As a part of our work over the past year, we have taken pains to document new systems and functionality. You can read the fruits of that here. In particular, there’s a LOT of new documentation related to javascript and soon some documentation on the blocks framework for registering PHP blocks.

As always, documentation is an ongoing effort and we appreciate any feedback you have to where we can improve.

The future

In the coming year we currently have planned (among other things):

  • Recurring Events Manager — an add-on which allows for easily managing recurring events which are predominately written in react.
  • Releasing more blocks for the new editor.
  • Beginning the work on refactoring our own editors to work similarly to the new WordPress editor except they’ll be more Event centric (vs. post-centric).

URL Validation in WordPress

If your WordPress website allows users to submit URLs, and you’re not sanitizing them properly, you could have a whole host of security problems. On the flipside, if you’re removing too much, you might not be allowing valid URLs either.

This issue is pretty complex, and there’s quite a bit of confusion surrounding it, but it actually has a really simple solution in WordPress.

tl;dr; take-away

If your WordPress website/theme/plugin accepts user input of a URL, you need to either:

  • sanitize it using esc_url_raw($url) before saving it
  • validate it using esc_url_raw($url) === $url , and if it fails validation, reject it

Don’t use filter_var($url, FILTER_VALIDATE_URL)!

A Problematic Issue

Apparently, determining what’s a valid URL in PHP is a struggle. And it has been for a while.

“Just use filter_var” It’s So Easy…

The “PHP approved” way of doing it is to use PHP’s built-in function filter_var($url, FILTER_VALIDATE_URL). That’s the most commonly accepted answer on Stack Overflow.So, problem solved right?

Actually, using filter_var has a number of problems:

    • it rejects URLs with underscores, eg http://my_site.com
    • it accepts URLs that could lead to Cross-Site-Scripting Attacks, eg http://example.com/">
  • it rejects URLs with multi-byte characters, eg http://스타벅스코리아.com

The problems with filter_var are explained better in this article, and discussed extensively on php.net’s documentation page.

Some have argued that filter_var is technically correct about what’s a valid or invalid URL. But really what we want is a **safe** URL, not just a technically valid one. And filter_var doesn’t do much to verify the URL is safe.

So that’s no good.

Just Make a Regex…

How hard is it to make a Regular Expression to validate URLs? Some poor soul asked that question once on Stack Overflow, and received a barrage of 19 answers. There was really no universally accepted answer (the most popular saying to use filter_var, and the accepted answer had other issues). Besides, I personally find regexes impossible to understand.

Just Find a Library to Do That…

Mika Epstein blogged about her struggles to validate a URL. She found a PHP library that mostly did it, but it still required some tweaking.

If you’re not using WordPress, you’re right to look for a pre-made library to do this, because it’s not straight forward…

I personally was quite unsatisfied that such a common task had no well-documented, good solution.

WordPress’ built-in Solution to Validating URLs

It turns out WordPress has a good option that’s super simple: esc_url_raw(). As documented here on wordpress.org, the function is meant to sanitize a URL before saving it to the database (not to prepare for outputting on the screen, that’s what esc_url() is for.)

Technically, the function is for sanitizing URLs (ie, removing bad stuff from them), not validating them (asserting whether or not they’re valid). But you can use it for validating like so:

If the url had nothing invalid in it, then it’s valid. Pretty simple eh?

And it works well too. None of the criticisms of filter_var apply to it. We ran it through some unit tests and I have yet to see any problems with it.

Invalid URLs according to esc_url_raw():

Valid URLs According to esc_url_raw():

A Better-Sounding, but Inferior, Alternative

There’s also a better-sounding function, wp_http_validate_url(). But from my testing, it found http://localhost invalid, when it should be valid. And it found URLs like  ​​​http://example.com/"<script>alert("xss")<script> to be valid.

The light documentation says this function is primarily meant for validating a URL for use in the WP HTTP API, not for storing a user-submitted URL. So although it’s name sounds better, it’s probably not what you’re looking for, unless you’re using the WP HTTP API.

Conclusion

esc_url_raw() function is used to ensure website URLs of commenters on WordPress websites are safe. Ie, it’s used to sanitize input from public users on websites running over 30% of the web, so it’s pretty battle-tested. If there was a security problem with it, or it was rejecting valid URLs, I’m pretty sure it would have already been discovered.

So is a URL valid? Just check esc_url_raw($url) === $url.

Thoughts on this?

URLs with Unicode Have Arrived to Event Espresso!

Want to put unicode characters in your event URLs? Eg Chinese characters like 中文, or even emojis like ? ? ?? As of Event Espresso 4.9.66 you can!
Before pull request 635, we were technically handling these characters correctly (we were percent encoding them), but it seems WordPress has a bug that was preventing these URLs from working properly. So we’re instead, technically, creating IRLs for now- that is, URLs with unicode characters in them. We checked the code, and WooCommerce and Custom Post Types UI plugins both do the same thing: whatever unicode characters you enter into the custom post type slugs get left intact (ie, not percent-encoded).

When you enter your the desired slug for the custom post type slugs (eg a replacement for events), we do at least run sanitize_title() on it, which makes sure it’s a string that can be used in URLs. So we’ll help event managers to not use totally invalid URLs.

So feel free to try to use any crazy characters you like in your URLs. Event Espresso will strip away the bad ones, and leave only valid ones. Just be aware that while putting emojis in your URLs might be fun, it’s also not a great idea to have potential attendees trying to manually enter emojis into address bars ?.

Penpals Explantion of PHP/cURL/openSSL/TLS/SSL Handshake Connectivity Issues

Today I spent a few hours wrapping my head around an issue where some websites weren’t able to connect properly to PayPal.com. I found it pretty tricky to understand because it involved quite a few technologies and programs:

  • PHP: a programming language, and a program to interpret lines of code written in that programming language (aka interpreter)
  • cURL: a program for composing messages to be sent over the Internet
  • openSSL: a program for encrypting messages according to one of many encryption algorithms
  • TLS (sometimes the same thing as SSL): a protocol for encrypting HTTP messages so they can be securely sent between two servers

And each of those programs has various versions, some which don’t play nicely with each other.

I like making analogies to the real world in order to better understand computer stuff. So here we go:

School Penpals Analogy

Imagine a gradeschool penpals initiative. Principal Helen Powers tells 5th grade teacher, Mr. Curls, to have the kids write letters off to another school somewhere else in the world. Mr Curls tells one of his students, Open-mouthed Susie-Lee Lewinski, to write a letter to another child on the other side of the planet. Here’s their pictures:

Except we don’t know what language the other child speaks (in fact, they haven’t decided where to send the letter yet). So the first letter Susie should send will find out what language the other child speaks, and if Susie can also write in that language (by the way, Susie is a polyglot: she speaks Arabic, Greek, Latin, Spanish, Esperanto and English).

So when everything goes smoothly:

  1. Principal Helen Powers asks Mr Curls to send a message to a given address
  2. Mr Curls tells Susie to compose a message to the unknown child
  3. Susie’s first letter is a “handshake” letter, briefly telling the other child what languages she speaks, and asking them which one they’d like to use.
  4. The other child, Charuky responds saying they’d like to speak in Esperanto
  5. Susie and Charuky continue to correspond in Esperanto

PHP, cURL, OpenSSL, and TLS Working Properly

What each character represents:

  • Principal Helen Powers: PHP interpreter
  • Mr Curls: cURL
  • Open-Mouthed Susie-Lee Lewinski: OpenSSL
  • various versions of TLS and SSL: different languages Susie can speak

When everything works smoothly,

  1. PHP code has instructions to use cURL to send a request to an IP address or domain name (like Principal Powers instructing Mr Curls to send a message to a physical address)
  2. cURL takes those instructions and, in turn, instructs openSSL to send a request to the IP address or domain name
  3. openSSL sends a handshake request, telling the other server what versions of TLS and SSL it can handle
  4. the other server responds with what versions of TLS and SSL it can handle
  5. a version of TLS or SSL is chosen, and subsequent messages are sent and received using that protocol

Problems in Communication

There are a few problems that can occur both between clients and servers communicating over the Internet, and between grade school penpals.

The Children Speak Different Languages

Or, the servers don’t have a shared supported version of TLS or SSL. In the case with PayPal, for security, they decided their server would only communicating using TLS version 1.2 or higher (and no version of SSL). This is fine for most websites on servers which have had openSSL updated in the last 5 years or so (specifically, servers should at least use openSSL version 1.0.1c/). However, if openSSL is too old, it won’t know how to communicate in TLS 1.2, and so PayPal will reply with an error like this:

The Principal Insists Susie Speaks in a Language the Other Child Doesn’t Know

Or, the PHP code specifies a specific version of TLS or SSL must be used, but the other server refuses to use that version.

Eg, in PHP code you can set curl_setopt ( $handle, CURLOPT_SSLVERSION,CURL_SSLVERSION_SSLv2 ) that will tell cURL to tell openSSL to only communicate using SSL version 2, not any other version of SSL or TLS. That’s fine if the other server is willing to communicate in SSLv2, but if they aren’t, like in the case of PayPal, they’ll respond with another error message.

Miscommunication between Mr Curls and Susie

Or, there is a bug in the version of cURL or openSSL in use, which causes openSSL to not auto-negotiate the best version of TLS or SSL to communicate with (the best usually being the most recent version of TLS they both support, or failing that, the most recent version of SSL they both support).

cURL version 7.29 had a problem that it supports TLS 1.2, but it needs to specifically told to use it instead of an different version of SSL or TLS. (This is like unless you tell Mr Curls that it’s ok for Susie to communicate in English, he’ll tell her it’s not ok.) According to this WordPress Trac ticket, the problem could be solved by specifically instructing cURL to use TLS 1.0 or higher (as of 2015, 99% of servers supported it) by using curl_setopt ( $handle, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1 );.

Also, cURL version 7.24 with openSSl 1.0.1e appears to have a different bug: if you added the above-mentioned line telling cURL to use TLS version 1 or higher, curl_setopt ( $handle, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1 );, openSSL wouldn’t communicate in TLS 1.2. (This is like Principal Powers telling Mr Curls “Susie is allowed to communicate in Spanish, Esperanto, or English” but Mr Curls misheard and missed the last two languages mentioned, and so thought Susie should only be allowed to communicate in Spanish). Ironically, if the PHP code doesn’t specify any SSL or TLS version (or specifies TLS 1.2), openSSL is able to auto-negotiate the best version of TLS to use.

The Dilemma We (Hopefully) Solved

In section “Miscommunications with Mr Curls”, you see that for cURL 7.29 we should add curl_setopt ( $handle, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1 ); to fix handshake issues, but for cURL 7.24 with openSSL 1.0.1e that actually creates issues. So what’s the best general approach (that works for most version of cURL)?

It was suggested you should just check which version of openSSL is in use, and if it’s a version that know how to use TLS 1.2, the PHP code should tell cURL to use that. But if not, the PHP code should tell cURL to just use the latest version of TLS possible.

But then, it was realized that the version of openSSL that PHP knows about may be different than the version of openSSL cURL is using, so it’s impossible for PHP to determine which version of openSSL cURL is using. Also, we’d still have the second “Miscommunication between Mr Curls and Susie”.

An important point though: for our currently purposes, it’s mainly communication with PayPal that was causing us grief.

So, our PHP code now specifically instructs requests going to PayPal to only use TLS1.2. If the version of openSSL in-use on the server doesn’t know how to use TLS 1.2, the communication will fail. But it’s doomed anyway: it doesn’t know how to communicate with TLS 1.2 and PayPal won’t accept anything else.

But, requests going to other servers, which might not yet support TLS 1.2, can fallback to openSSL’s normal TLS/SSL-version auto-negotiation, which get it right 99% of the time.

What’s more, we saw the most popular e-Commerce WordPress plugin is doing exactly that too. So we’ll be in pretty good company anyway.

See a Problem?

Do you see a problem in my logic? Or any parts that don’t make very much sense? Please let me know in the comments!


This post originally published on my blog.

Troubleshooting iPhone App Communication with Fiddler

This post is also on my personal blog.

While troubleshooting an iPhone app, it’s sometimes helpful to see exactly what HTTP requests its sending.

Enabling Fiddler to View iPhone HTTP Traffic

I managed to do that with Fiddler by following these steps: https://www.pluralsight.com/blog/tutorials/using-fiddler-with-an-iphone-ipad

I found my laptop’s IP by doing this:

  1. Click on the Start menu and type cmd. When you see the cmd applications in Start menu panel, click it or just press enter.

  2. A command line window will open. Type ipconfig and press enter.

  3. You’ll see a bunch of information, but the line you want to look for is “IPv4 Address.” The number across from that text is your local IP address.

from https://lifehacker.com/5833108/how-to-find-your-local-and-external-ip-address

I actually saw a bunch of lines for “IPv4 Address”, one for each wireless adapter (I have a bunch of adapters running: a WiFi one, an ethernet one, a bunch of virtual host ones for my virtual . In my case, I wanted the WiFi one, because that’s how I was connected to the network.

Once I had that all setup, I could see my phone’s HTTP traffic in fiddler… but not HTTPS traffic.

Enabling Fiddler and iPhone to work with HTTPS

For that I generally followed the steps from https://www.telerik.com/blogs/using-fiddler-with-apple-ios-devices, including the part about downloading the Certificate Maker (although it was important that when I did that, I had fiddler configure to NOT decrypt HTTPS traffic yet, as I think that interfered) especially the section on “Decrypting HTTPS Traffic from iOS”.

But the part where it said to visit “ipv4.fiddler:8888” didn’t work (safari couldn’t find it). So instead, from my computer, I went into Fiddler, then Tools, then HTTPS tab, then enabled “Decrypt HTTPS traffic”, then to “Actions” and “Export Root Certificate to Desktop”. Then I emailed that certificate file to myself, opened the email from my phone, opened the attached certificate, clicked and clicked “Install”. So that installed the certificate on my phone, but didn’t enable it yet.

To enable the new certificate, I went into my phone’s Settings app, then General section, then About, then scrolled to the bottom where it said ” Certificate Trust Settings”, saw the new certificate listed but not enabled, so I enabled it.

Viewing iPhone HTTPS Traffic from Fiddler

After that, all HTTP traffic from my phone appears in Fiddler. Like you can see in this screenshot, there is a slightly yellow area in the middle-right that says “Response body is encoded. Click to decode.” Clicking that reveals the decoded message. (I would have thought that should have been “Response body is encrypted. Click to decrypt”.)

Conclusion

Using Fiddler, I was able to watch the HTTP and HTTPS traffic from my iPhone (I previously tried to use WireShark in promiscuous mode, but my iPhone traffic never appeared in there).

This was a little handy while debugging an issue we were having with our Event Espresso mobile apps… except once I had this all setup, it seems getting my iPhone to trust the root certificate generated by the Fiddler Certificate Maker caused the error to go away… pretty annoying, but it isolated the problem pretty well: either our problem was with the website’s certificate, or in the mobile app code that handled the certificate.

Anyways, let me know if you have better suggestions on how to debug mobile app HTTP communication from a computer!

Line Ending Troubles with Git, a Virtual Machine, and Windows

This post originally from my blog.


This is some technical stuff I learned today using Git, VVV with a Linux virtual machine, and Windows.

I recently started getting these annoying messages again whenever I’d commit anything using Git:

warning: LF will be replaced by CRLF in payment_methods/Paypal_Standard/help_tabs/payment_methods_overview_paypalstandard.help_tab.php.

The file will have its original line endings in your working directory.

I had recently played around with my code editor’s settings (PHPStorm), so that’s probably why.

I mostly ignored these for a while, but I found I couldn’t apply patches from WordPress core without having to run dos2unix on each file. Eventually I finally broke down and tried to figure out what happened, and thanks to my two co-workers, Brent Christensen and Darren Ethier, I think I finally have it figured out what was my problem, and a pretty good solution.

My Setup

I actually previously blogged about my development setup, but here’s the gist:

  • my computer’s operating system is Windows 7 (ie my “host machine” is Windows).
  • I use VVV, which uses a virtual machine (when I type in the URL of a website I’m developing locally, its this virtual machine that handles the request; ie my “guest machine” is Linux).
  • Also importantly, VVV magically copies certain folders from the guest Linux machine to my host Windows machine. This is convenient so that the guest Linux machine can use those files in order to serve web pages, but I can read and edit those files from my host Windows machine.
  • I use Git from my host machine, and sometimes run commands from within the guest machine.

Like was suggested on GitHub’s help page about line endings, I had the git configuration setting core.autocrlf true.An Important Note on Line Endings

On Windows, each new line in a file is actually represented by the hidden characters \r\n a.k.a. “carriage return, line feed” characters.

Whereas on Mac and Linux machines, new lines in files are represented just by \n, ie, *just* a new line character.

So, by default, my host Windows machine adds \r\n, but my guest Linux machine adds \n. So if I edited a file using Windows it would add one type of line ending, but then when I edited the same file from my guest Linux machine, it replaced that line ending with a different one. So there was a bit of a tug-of-war going on.

Git’s core.autocrlf Setting

They say

The git config core.autocrlf command is used to change how Git handles line endings. It takes a single argument.

On Windows, you simply pass true to the configuration.

Sounds simple enough, set that setting to true then all your problems are solved.No need to worry about line endings. Well, those warnings I was getting showed otherwise.

What core.autocrlf Actually Does

When you tell Git to use core.autocrlf true, you’re telling it that when Git checks out a file, regardless of whatever line endings are in the file in the original repository, it should replace those with \r\n. So when viewing the file, all the line endings are \r\n. But when you commit the file and then push it back to the original repository, Git takes care of changing those \r\ns back into \ns, which the original repository wants.

Using a Linux Virtual Machine Means You’re More Linux-y than Windows-y

Theoretically, this is what every Windows user wants, because Windows will see its favourite line endings, \r\n.

But the problem was: I’m not just a Windows user. I’m also using Linux, in the guest virtual machine. And like I described earlier, the host Windows machine was always adding \r\n, but the guest Linux machine would replace that with \n.

What’s more, the only program on my host Windows machine that ever reads or edits the files is PHPStorm, which handles \ns just fine. So it really isn’t of any advantage to me to tell Git to use \r\n when it checks files out from the repository. It just causes a tug-of-war.

My Solution

My solution was to tell Git to use core.autocrlf input. This tells Git to leave the line endings alone when checking out (so usually the original repository used \n, which I want), and only when committing to make sure the files being sent use \n.

This way, even when I checked out files from my host Windows machine, it would leave the line endings as \n, which my guest Linux machine was also happy with. No more warnings. No more Git Diffs showing only a line ending change. WordPress core patches apply cleanly. Much better.

One Gotcha in Application

So what if you’ve already checked out a Git repository and it’s filled with /r/ns? The setting I described only works when you checkout a file, which Git avoids doing unnecessarily. The workaround I found was to create a new branch, delete everything in it (except the hidden .git folder), make a commit (maybe that’s unnecessary, but it’s what I did) and then re-checkout the original branch. That will force Git to checkout all the files, which will now all have the desired /n line endings.

Conclusion

GitHub’s recommended setting for core.autocrlf truefor Windows isn’t the best option if the repository in use is actually in a folder that’s copied to guest Linux virtual machine.

Using core.autocrlf input allows the host Windows machine and guest Linux machine to play much nicer together.

If you see a better way to avoid having my host Windows machine play tug-of-war with my guest Linux machine over line endings, please let me know!

Fixing a PHP DateTime Bug

In version 4.9.59p of Event Espresso Core, there will be a workaround fix for a PHP DateTime object bug we discovered while troubleshooting an issue on our EventSmart platform.  I just thought I’d write a few words about this bug because it’s entirely possible other developers will encounter it in the wild as well.

On Event Smart we had a customer start reporting that outgoing message notifications from our messages system were displaying times for the related events that did not match the dates setup for the event in the event editor.  For instance, the time configured for the event was 4am-1pm but the time in the message was 12:00am-9am.  The site was set with a timezone of UTC-4, so on first glance this appeared to be something wrong with our timezone conversion logic because the difference between those times was 4! However, with dates and times being such a critical part of our application, we have pretty significant automated test coverage of all our logic related to that including coverage of the code used in our messages system.  The other puzzling thing was that the messages system was using the same internal methods as elsewhere in the app for displaying those dates and times and those places were displaying things fine. Hmmm… puzzling.  Further, we noticed that we were only able to reproduce this on sites that were using a UTC offset instead of a timezone string for their timezone.

Then I remembered that all outgoing messages and notifications are being processed by separate workers via a gearman job system so it’s possible there’s something in that environment that is subtly different than what is happening in a normal web request.  So I began adding a bunch of logging to capture time output within the environment of the gearman workers themselves when messages were being generated.  At first I thought, maybe there’s some caching that is not getting cleared correctly but that was ruled out.  And then I started seeing some weird behaviour where right after changing a timezone on a DateTime the internal unix timestamp in the DateTime object was affected !  That’s significant because the unix timestamp should not change.  It’s the primary reference for that DateTime object!  Then I thought I’d see if I could reproduce this with just some standalone code, something like this:

I ran this code on 3v4l.org (super nifty tool for running php code across multiple php versions) and lo and behold look at the results!

 

This means that there’s version specific behaviour for PHP’s DateTime object!  Once I realized this was going on it immediately clicked that on EventSmart it’s possible we might have different versions of php being used in different contexts and sure enough we discovered that our host had forgot to have our gearman workers use the latest version of PHP7 and they were still using PHP5.6 where web requests were using the latest version of Php 7.  Once we changed the PHP version for our workers boom, things started working again.

So I realized, this would affect all our Event Espresso customers as well (since Event Smart is built using Event Espresso), and I began to dig more.  In the process I discovered that this actually was a bug reported and fixed by PHP .    I also discovered, that there is a workaround.  If you use the DateTime::getTimestamp() call immediately after invoking DateTime::setTimezone(), the internal unixtimestamp will be reset to it’s correct value.  So the workaround I’ve added for the next version of EE core to be released is to simply implement that wherever timezone is changed.

A Summary of the Bug

  • Affects PHP versions 5.5.10-5.6.x, 7.0.0-7.0.16, 7.1.0-7.1.2 – PHP 7.2 branch is not affected and earlier versions are not affected because earlier versions of PHP don’t have the ability to setTimeZone with a UTC offset.
  • Only happens when calling setTimezone() with a UTC offset.
  • Only demonstrates itself when immediately after changing the timezone using setTimezone one uses format('U') to get the unixtimestamp.  In my testing if you used getTimestamp() to get the unixtimestamp you’d get the expected value.

Upcoming Change to `DateTime` objects stored in EE model entities

Currently in Event Espresso core, we store datetime information for fields in our model entities that are described by EE_Datetime_Field as DateTime objects.  This makes working with DateTime objects easier.  You can read more about when we started using DateTime objects internally in this post.

Recently, we uncovered a potential flaw with this implementation in that this internally stored DateTime object was exposed via the EE_Base_Class::get_DateTime_object method and thus directly mutable by client code.  We don’t want this to be mutable because it can lead to unpredictable and unexpected behaviour for any other code working with that entity in the same request.

Our intention has always been that the internal DateTime object stored in the EE_Base_Class entity would be immutable externally and any mutation would have to occur through the exposed setters.   Besides the exposure of the instance via get_DateTime_object we also missed the fact that if a EE_Base_Class child instance (eg. EE_Datetime) was cloned, the clone would receive the original instances of any cached objects on properties.   Thus in the case of cachedDateTime objects, when client code did something like this:

 

The change on the clone start date would also affect the date on the original $datetime instance because it’s mutating the same DateTime instance.

So beginning with version 4.9.59.p of Event Espresso core, the internal DateTime objects become immutable outside of the EE_Base_Class entities and are only mutated through setters.  We’ve changed things so that:

  • EE_Base_Class::get_DateTime_object returns a clone of the DateTime object.  You can mutate that clone and it will not affect the original in the entity.
  • Utilizing the __clone magic PHP method, we’ve ensured that anytime a EE_Base_Class child instance is cloned, that any internal DateTime objects are also cloned.  Thus with our example above, the $clone instance can have its date and time information modified and it will only affect the clone, not the original.

In all of Event Espresso core and our add-ons we only had a couple places where the DateTime object was being mutated directly outside of a EE_Base_Class and we modified those places so it would work with the changes.  Although we believe its unlikely it’s happening, the purpose of this blog post is to alert any developers expecting to be able to directly mutate these DateTime objects to make sure you understand this is will no longer be possible.  All mutations must be done through the exposed interface.

Things Learned in WordPress Support Forum Today

Recently, a old friend of mine said, on Facebook, that they were having issues with their WordPress website. I volunteered to help them, thinking I’d probably learn something in the process.

The Problem

My friend’s problem suddenly appeared a week or so: when they updated a post, they would see a 404 error. (Here is a video, showing me saving a post, and then getting a 404 error). My friend asked for help from their host, and they suggested she should first update WordPress (she was on 4.8.3 or something), but for some reason her automatic updates also weren’t working.

So there were really two problems:

  1. Automatic updates weren’t working on their website
  2. They were getting 404 errors when saving posts

The Solution

Automatic Updates

My friend sent me credentials to log into their site, and strangely enough, her site reported that WordPress 4.8.3 was the most recent version (at the time of writing, the current version of 4.9.4).

When I went to turn on WP_DEBUG in their wp-config.php file, I noticed define( 'WP_AUTO_UPDATE_CORE', false ); was set. I also noticed they had a cPanel plugin active, which automatically added that, and even changes WordPress core files in order to handle updating WordPress via cPanel. While that sounds ok, unfortunately its corresponding cPanel cpAddon was deprecated and so doesn’t take care of updating WordPress either.

So I removed that line in their wp-config, and disabled the cPanel plugin. Then WordPress could auto-update as normal. Yeah! But that didn’t fix the 404 error when saving posts.

404 Errors when Saving Posts

This issue was much trickier to track down. I traced through the code and determined:

  • the requests were NOT making it to the wp-admin/post.php file, but were instead going to index.php (this is normal when mod_rewrite is on and you’re sending a request to the site’s front-end, but unusual for requests to the admin)
  • this issue was reproducible even with all plugins deactivated
  • the issue was reproducible on a brand new install (on their server) of WordPress, using the default theme
  • when I turned WP_DEBUG and WP_DEBUG_LOG on, there were no additional errors (there were some warnings on some other pages, but nothing related to this)
  • the apache server logs reported nothing of interest (they reported a 404 error, which I was also seeing, but nothing more)
  • the .htaccess file was unaltered
  • when I copied the website’s exact files to my local vvv setup, I couldn’t reproduce the issue

Given all the above, it seemed the problem wasn’t being caused by a plugin, theme, or even WordPress core. I spoke with the hosting company’s support, but they were basically trained to assume all problems were WordPress’ fault.

I posted an issue to the WordPress support forums and to WordPress Stack Exchange to see if I was overlooking something. During that process, I realized an important clue: if I set permalinks to default, instead of getting a 404 error, I would get a different error

Forbidden You don’t have permission to access /wp-admin/post.php on this server. Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request

https://drive.google.com/a/eventespresso.com/file/d/14Kzkm14AJ5e7XXKCAuR-4NHBlcOZKdJ_/view?usp=drivesdk

At first I skimmed over this, thinking it was just a different kind of 404 Not Found page. But later I realized this was the crux of the issue: access was being denied to wp-admin/post.php, and so somehow Apache was seeing if it could handle the request by sending it to index.php and then using URL rewrites… or something, I guess I need to freshen up more on how URL rewrites work.

Anyways, also important: there was also an error while recording the error, so there was no trace of it on the server or in logs.

I approached the host’s support folks with this info, saying all signs pointed to mod_security having a bad directive. They responded they would not deactivate mod_security for one customer (reasonable) but they also weren’t going to make any attempt at verifying what I was reporting. This was the first report they’d received of it, so obviously it must be an isolated incident.

I would say their position was reasonable, but with an important exception: they had not tested any other WordPress sites on their server with their mod_security rules. There were other customers using WodPress, but the host didn’t want to test on them, nor did they want to set up WordPress on one of those sites and test it themselves. So they had no proof the error did not exist, and so were  taking that as proof the error did not exist.

What’s more, I pointed out this error came up somewhat recently, and only occurred when saving posts with hyperlinks in them, so it’s possible others’ just hadn’t noticed yet.

Anyways, the solution seemed to just be to change hosts.

 

Incidental Information

I didn’t realize that before I created the WordPress support forum post, my friend also created one. Once I realized this, I thought I’d point out my mistake to the forum moderators, because maybe they would want to mark my new forum thread as a duplicate. Instead, they kinda censured me for “jump[ing] into other topics.” Either my comment wasn’t very clear, or they only skimmed it, but either way they thought the duplicate issue I was reporting was another topic…  so obviously there was some miscommunication there.

Also, a little way into my original forum thread, the user who was helping me out said they couldn’t see any of my screenshots or videos (which I thought were really helpful) because they were blind! OUPS. To be honest, this was probably my first interaction, ever, with someone online who’s actually blind. This made accessibility issues much more real, for me.

 

The Take-Aways

So here’s what I learned from participating while helping my friend out:

  • mod security. Pretty standard tool for adding security at the webserver level for Apache. Also a possible source of problems when being too aggressive.
  • it really does help to have a host who knows/specializes in WordPress. When I saw a host “specialized” in WordPress, I used to always think that was rubbish. WordPress is easy, why do I need a special host for running it? The answer is that, it seems, is that many hosts aren’t setup quite correctly for running WordPress, like my friend’s host. What’s more, many of them utterly refuse to help once you mention “WordPress”. Pagely has been very helpful in that regard for us in running EventSmart.
  • helping folks for free can often builds confidence, which can build business. Jackie McBride helped troubleshoot the issue with me, and once my friend was again in the market looking for a new host, Jackie was one of the people my friend was probably going to buy hosting from. That might actually be Jackie’s purpose in “volunteering” in the support forums: acquiring new customers by first helping them out, building a relationship of trust, and then it’s pretty easy to transition to a paid relationship. And I don’t think I mind that: I prefer that to receiving spammy emails or advertisements

 

Escaping Problems with Slashes in WordPress

If you’re a web developer who uses WordPress, you’ve probably been surprised to find extra backslashes magically added to the $_GET and $_POST request data, right? It’s a confusing situation, with information dispersed sparsely across the internet. While working on Event Espresso, we’ve had our share of troubles with it, and wanted to share what we’ve learned.

If you get “slashes wrong” in your code, you can end up with really big security problems (specifically, SQL injection problems), bugs, or find user-created content gets litterred with extra slashes like I\\\\\'m \\\\\'drowning\\\\\" in slashes!.

So, do you know how to work with slashes? Here’s a quiz:

  1. Consider this line of PHP:echo 'it\'s';. The \ is an “escape character”, and \' is an “escape sequence”. True or False?
  2. The HTML <input value="\ \\ \r\n \' \""> will be displayed as . True or False?
  3. The following 3 lines of PHP all show the exact same thing: echo '\ \\ \r\n \' \"';, echo "\ \\ \r\n \' \"";, and ?>\ \\ \r\n \' \"<?php . True or False?
  4. The PHP code echo addslashes('billy \"the nose\"') will echo billy \\\"the nose\\\". True or False?
  5. Adding backslashes to request data is the recommended way to prevent SQL injection. True or False?
  6. Your WordPress code can rely on $_GET and $_POST request data always having backslashes added to it, regardless of whether PHP’s “Magic Quotes” is turned on. True or False?
  7. $wpdb->prepare, $wpdb->insert and $wpdb->update all take care of adding slashes, so data provided to them should not have slashes added to it. True or False?
  8. Calling $_POST = stripslashes_deep($_POST); is the best way to handle backslashes added to request data. True or False?

Each of the following sections will contain an answer to one of the questions.

Slashes Explained

Escape Characters and Escape Sequences

Consider this line of PHP:echo 'it\'s';. The \ is an “escape character”, and \' is an “escape sequence”. True

Wikipedia says

An escape sequence is a sequence of characters that does not represent itself when used inside a character or string literal, but is translated into another character or a sequence of characters that may be difficult or impossible to represent directly.

For example, in this PHP echo 'it\'s'; the \' is an escape sequence. Because the string started with a single-quote, we can’t add a regular single-quote to the string without abruptly ending it. echo 'it's'; would be invalid PHP, because there’s a valid string it, but then there’s an unexpected s and single quote. So if you want to put a single-quote in your string, you need to use the escape sequence \' to represent that.

And by the way, in the string \', the backslash is acting as an escape character, meaning the character(s) after it has a special meaning. Backslashes are often used as escape characters in PHP strings and MySQL queries, but not in HTML.

HTML and Backslashes

The HTML <input value="\ \\ \r\n \' \""> will be displayed as . True

When displaying your page’s HTML, Web browsers leave backslashes alone and don’t treat them as escape characters.

So if there is \ \\ \r\n \' \" in the page’s source code, that’s also exactly how the browser will display it to the user.

That goes for inside HTML input attributes too: backslashes are not escape characters. So

<input value="\ \\ \r\n \' \""> will show this

notice the double-quotation mark disappeared. That’s because \" wasn’t considered an escape sequence, so the " was interpreted as a normal double-quote. (And so that’s actually bad HTML because there is now an extra double-quote in that tag).

So clearly, adding slashes in front of quotes inside HTML attributes doesn’t escape them. If you want to display a double-quotation mark inside an HTML input’s value, you should use the HTML entity &quot;

PHP and Backslashes

The following 3 lines of PHP all show the exact same thing: echo '\ \\ \r\n \' \"';, echo "\ \\ \r\n \' \"";, and ?>\ \\ \r\n \' \"<?php . False

A backslashes can mean different things depending on whether they appear in a single-quoted string, double-quoted string, or are fetched from the database.
In a single quoted string, they’re only treated as an escape character if they appear in front of a single-quote or another backslash.

So echo '\ \\ \r\n \' \"'; will show \ \ \r\n ' \". Notice the backslashes are all interpreted as literal backslashes EXCEPT

  1. \\ became \
  2. \' became '

In a double-quoted string, they are treated as escape characters when placed in front of many other characters (see the table under “Double quoted” on the PHP page), but in front of double-quote instead of a single quote.
So, echo "\ \\ \r\n \' \"" will show \ \
\' "
. This time, many more of the backslashes were considered escape characters, but not all, and not all the same ones. Specifically,

  1. \\ became \ (same as with a single quoted string)
  2. \r became a carriage return
  3. \n became a new line
  4. \" became "

Notice the \' was NOT considered an escape sequence, like it was with a single quote string.

Lastly, backslashes are also treated differently if they’re not from a string in your PHP code, eg if they’re from a database column or HTML code entered outside of PHP tags. In this case, they are never treated as escape characters. So,?>\ \\ \r\n \' \"<!--?php , and global $wpdb; echo $wpdb->get_var('SELECT option_value FROM wp_options WHERE option_name="my_option"');, where the option_value is \ \\ \r\n \' \", will both show \ \\ \r\n \' \". Ie, none of the backslashes were considered an ecape character.

Magic Quotes and Backslash functions

The PHP code echo addslashes('billy \"the nose\"') will echo billy \\\"the nose\\\". True

From php.net, addslashes

Returns a string with backslashes before characters that need to be escaped. These characters are single quote ('), double quote ("), backslash (\) and NUL (the NULL byte).

So code echo addslashes('billy \"the nose\"') will echo billy \\\"the nose\\\", because each backslash in the original string gets another one added, and each double-quote gets a backslash added in front of it. Each time addslashes is called, more slashes get added. So echo addslashes('billy \\\"the nose\\\"') will echo billy \\\\\\\"the nose\\\\\\\", etc.

(Side note: because we’re talking about slashes and WordPress, you may like to know about WordPress’ wp_slash and wp_unslash functions. They’re mostly the same as addslashes and stripslashes, except they work with either a string or array; and it’s possible they may diverge more in the future.)

SQL Injection and Extra Backslashes

Adding backslashes to request data is the recommended way to prevent SQL injection. False

Adding slashes to a string before using it in MySQL code can save you from SQL Injection, so some PHP developers a long time ago came up with the idea of always adding slashes onto request data received from the user, eg $_GET and $_POST data. This feature was known as “Magic Quotes”.

PHP’s Magic quotes was an attempt to make GET and POST data safe, by default, for use in database queries. Consider the next line of code

If $_GET['title'] were ";DROP TABLE wp_posts;-- this would be SQL injection because the generated SQL would be

Which would delete the entire posts table! So that would be a big problem if $_GET didn’t get slashes added onto it. But if we called wp_slash on $_GET['title'] the SQL generated would have instead been:

Which would instead look for posts with the title ";DROP TABLE wp_posts;-- which is harmless.

So, the original purpose of magic quotes was good. But it turns out that’s not the only thing that you should do to use input before using it in your database- what you do to it depends on what type of database you’re using, because MySQL escapes sequences are different from Postgre escape sequences, etc. So in reality, you usually need to remove extra slashes and then use a database-specific escaping function. Needing to do this was obviously a pain, and so PHP 5.4 and higher officially no longer support magic quotes.

WordPress, Superglobals, and Backslashes

If you incorrectly think a string has backslashes added onto it, and then directly use it in an SQL query, you can have SQL injection problems. But conversely, if a string does have slashes added onto it, and you add backslashes to it again, users will see extra slashes added all over. And because not all WordPress users had Magic Quotes enabled, it was difficult to know if request data had slashes already added or not. So WordPress core developers tried to at least make things consistent by always adding backslashes onto request data regardless of whether Magic Quotes were enabled or not, even for versions of PHP that don’t even support Magic Quotes. (Documentation on this is pretty sparse, but read the notes on stripslashes_deep if you want more info.)

Your WordPress code can rely on $_GET and $_POST request data always having backslashes added to it, regardless of whether PHP’s “Magic Quotes” is turned on. False

So because WordPress always adds backslashes to the request data, you can rely on there being backslashes, right? Wrong. Despite it being a bad practice, many plugins and themes have $_POST = stripslashes_deep($_POST); which removes backslashes again, for all other code too, but only if that particular plugin is active. So watch out!

What’s more, even if there are no plugins or themes interfering with slashes, the request data might not have slashes added onto it yet. WordPress only adds the slashes after it has done the plugins_loaded action (inside wp-settings.php it calls wp_magic_quotes). So if you have code that’s using request data before or during plugins_loaded, its request data will NOT have slashes added onto it by WordPress; but any code after plugins_loaded, like code running during sanitize_comment_cookies or afterwards, will have slashes added onto it.

WPDB and Backslashes

$wpdb->prepare, $wpdb->insert and $wpdb->update all take care of adding slashes, so data provided to them should not have slashes added to it. True

When interacting with the database in WordPress, you’re best to use the global $wpdb class. Among others, it has many methods for interacting with the database, but it’s important to know which ones expect your input to be escaped and which ones do not.

When passing data into its get_results, get_var, get_col, and query, its expects that the string you’re passing into them was already prepared for use in the database, otherwise it could introduce SQL injection problems. Providing slashed request data helpers prevent SQL injection, but it’s not the best way to prepare request data for use in a database query because there are other escape sequences it might miss.

WPDB’s prepare is the preferred way to prepare request data for use by one of those previously-mentioned functions. Here’s a link to its documentation. But an important aspect not mentioned in the documentation is that the arguments being passed into it, if from request data, should not have backslashes added onto them. That’s part of what prepare does.

E.g., if $_GET has had slashes added onto it, this is how you should use it with $wpdb->prepare

Because if $_GET['title'] had the value of What I think about \"Star Wars\" (notice the extra slashes, which would have been added by WordPress, and not present in the user’s original request data), we want to send What I thnk about "Star Wars" to the database, without those pesky extra slashes. That’s why I added wp_unslash to that snippet.

There are other methods on WPDB that also take care of preparing the data you send them. Specifically, insert, update, and delete. You don’t need to call prepare on input for them because they assume the data you’re providing them hasn’t been prepared, nor had slashes added onto them.

So long as you use WPDB’s methods that prepare the data for use in the database (and avoid calling it repeatedly on the same string, which is another gotcha), you don’t have to make sure the request data has slashes added to it. And if, by mistake, you call wp_unslash too many times on a string, you will probably irritate users because their backslashes will disappear on their submitted content, but SQL injection will not be a problem for you. (While neither calling wp_slash or wp_unslash too many times it good, wp_unslash will probably be less annoying for users. If you call wp_unslash too many times, users’ backslashes will disappear, but how often do folks use backslashes in their submitted content, anyway? Whereas if you call wp_slash too many times, slashes will appear in front of single and double quotes, which are characters used far more often.)

Your Code and Backslashes

Calling $_POST = stripslashes_deep($_POST); is the best way to handle backslashes added to request data. False

Changing the $_POST globally, for all other plugins, can lead to plugin conflicts. If another plugin does that too, then you will be removing backslashes the user intended to add. It’s best to not interfere with how other plugins are handling the backslashes.

If you’re making a small, simple plugin or theme, here’s what I’d suggest: during the plugins_loaded action, before any of your other code runs, create a copy of $_GET, $_POST, $_REQUEST, and any other request superglobals you will want to use. Make those copies available globally, and use them instead of the PHP superglobals.
(Note: originally suggested copying the superglobals right away, but I realize that isn’t very friendly to other plugins which may want to modify your plugin’s behaviour, and it’s bad form for a plugin to do any actions before init action. See this video from WordPress.tv for me.)

E.g., here’s a plugin’s code that copies those global variables into its own global variables

In the above code we know $my_get['username'] doesn’t have slashes added onto it because we created that global variable before WordPress added slashes onto $_GET, so we don’t need to call wp_unslash on it. Also, because WPDB’s prepare takes care of escaping the input, we don’t have to call wp_slash nor do anything else to prepare it.

Event Espresso and Backslashes

In Event Espresso, that’s more-or-less what we’re trying to do. During the plugins_loaded action, we create a object called EE_Request which stores the request data separately BEFORE WordPress has added slashes onto it. We make that object available from a singleton. And in the rest of our code, we can use that EE_Request object, instead of $_REQUEST, and know it reliably does not contain extra slashes. When we go to use request data in database queries, we make sure to prepare the data using WPDB’s prepare method so we can protect from SQL injection.

Although, in full disclosure, we are working to avoid using the singleton because both singletons and global variables are considered bad design in the programming world. We are working to rectify this sub-par code by instead using dependency injection, but it will take time.

Also, at the time of writing, admittedly we still use $_GET and $_POST directly, although we’re working to remove those. (Part of the purpose of writing this post was to think through the problems of that. We also plan to totally avoid using the request superglobals in the future.)

If you have created code that integrates with Event Espresso, we highly recommend you also start using EE_Request instead of $_GET and $_POST directly. E.g., $my_var = isset($_REQUEST['my-key']) ? $_REQUEST{'my-key'] : null; can be replaced with $my_var = EE_Registry::instance()-&tl;REQ->get('my-key', null). Failure to do so means you may get fall into the slashes trap!

Summary

In order to avoid the tangled mess of slashes in request data in WordPress, create your own copy of the request data before WordPress adds slashes to it, and use your copies instead. Also, be sure to always prepare that data using WPDB’s helper methods, like prepare before using it in the database.

And don’t be fooled by slashes again!