Author: Ho Yin Cheng
/
Created: February 13, 2018
I’ve forked and made minor pull requests in the past, but this past week (a full 3 days of analyzing/coding & 1 day of documentation), I made my first real fork of an existing project. It was quite the task as I had to learn the ins and outs of the existing codebase in order to make the large changes I wanted. Today, I’ll be talking about my entire thought process as I forked google-maps-services-js into react-native-google-maps-services.
I wanted to use the Google Places API, specifically Nearby Search, in the React Native app that I’ve been building and mentioning in my devlogs. The problem was that I was unable to find a satisfactory client library to use (something I’ll talk about later). So, I was left with the choice of either writing one from scratch or forking an existing repository and changing it to suit my needs. In the spirit of open source, I decided that forking was the way to go (it’s also a lot less work ;).
When people talk about Google Places API, there are actually several different parts to the API serving different functionality. This can get confusing when you search for information on how to use the API as you can get conflicting information as each has somewhat different functionality. It also gets very annoying when looking for client libraries as all of them use a generic name of “Google Places” but are actually built for only some of the APIs.
On top of that, the Places API is a subset of the APIs groups under the Google Maps API name. This can cause a lot of confusion when looking for information on how to use the different libraries.
Then on top of that, there are slightly different APIs exposed by the Android and iOS libraries. Both of which can be accessed by React Native if you choose to go the native module route.
Lastly, on top of that(!), there are different APIs exposed by the Places Library in Maps JavaScript API and others. There’s a lot to dig through and it’s made all the more confusing by the reuse of names. For my purposes, I’ll only dive into the web services APIs and a bit into the mobile APIs.
This was a multi-day effort of reading docs, diving into source code for poorly documented projects, and testing integration with React Native. This is what I discovered:
Official Client Libraries exist for Java, Python, Go, and Node.js.
If you want to try installing it into React Native, you can use the following libraries to get access to the Node.js core libraries (required since they aren’t part of React Native’s JavaScript engine; see also):
Installing the individual core libraries needed and using babel-plugin-rewrite-require to transform the requires that don’t have direct naming equivalents. And you’ll also have to install the q promise library for JavaScript as that is used in the code but no error is thrown when it isn’t found. Here’s an example of what you can put into your .babelrc:
|
|
The core bug is this:
YellowBox.js:82 Possible Unhandled Promise Rejection (id: 0):TypeError: Cannot read property 'getReader' of undefinedresponse.body’s getReader() method. However, there is no body property of the response. The question is why? There is a response with data. Just no body (undefined).fetch implementation lags behind the official as can be seen by the dependencies:
"whatwg-fetch": "^1.0.0","node-fetch": "^1.3.3",whatwg-fetch lags behind the current release of whatwg-fetch (2.0.3).fetch to be replaced by whatwg-fetch by doing:
yarn add whatwg-fetchwhatwg-fetch’s source code to not check for self.fetch (fetch.js line 4)whatwg-fetch. Previously, it didn’t because that check prevented whatwg-fetch from replacing the native fetch call. Which conflicts with how RN is checking for whatwg-fetch (react-native/Libraries/Network/fetch.js and react-native/Libraries/Core/Devtools/symbolicateStackTrace.js). Yay for conflicting code bases!body remained undefined.fetch response’s body.react-native-google-places-autocomplete
Nothing works.
Before going down the route of writing or forking a client library, it’s prudent to examine your original project needs to see if this is the best use of your time. And so, that’s what I did.
Some suggestions and comparisons to read:
These are the best of the best. Foursquare is the most attractive offering. Mapbox is great too as it would just be a part of their total offering (you can maybe even remove Google Maps completely). Agolia is a big name, but I’d rather go with Foursquare if I weren’t to try and use a completely Mapbox-only solution. Lastly, OSM is included because it’s the one self-hostable solution.
While these certainly do provide APIs that have the functionality I desire, none have an official React Native client library. Foursquare comes the closest but the libraries do not look very well maintained. As such, I felt that my time was better spent going with Google for now as it has the most robust looking code base for me to work off of.
If in the future I decide that I want to remove all Google services, I can probably reuse what I’ve learned from their codebase to write a more robust client library for Foursquare/Mapbox/etc.
With that out of the way, I made my decision to fork google-maps-services-js. I had to fork because:
makeUrlRequest (#88 [#64(https://github.com/googlemaps/google-maps-services-js/issues/64)]). However, this wouldn’t work because we wouldn’t be able to remove the dependency on url. It would also be more difficult to differentiate between put and get requests because of how the architecture of the code makes assumptions in makeApiCall based on where it places value.makeUrlRequests even easier to create. This would allow swapping out axios much easier in the future. Passing around the config options in a sensible manner and making them available for assembly in the request.
requestUrl in makeApiCall, but it would remove the ability to swap out backends. I am also unsure of how reusing axios instances behave.Given that rationale, it was an easy decision to make. Furthermore, Github makes forking so easy that it seemed like a no brainer over trying to write from the ground up. Do keep in mind that while you work on your fork, the upstream repository may get some commits pulled in that you need to merge into your fork. I found these guides to be the most useful in understanding how to do this and how to write my commit messages:
These are the parts of the google-maps-services-js source that use core Node.js modules:
google-maps-services-js/lib/index.js
util.deprecate() to mark deprecated API methods.google-maps-services-js/lib/internal/cli.js
Process to parse args.google-maps-services-js/lib/internal/make-api-call.js
Url used to create the URL to send and to generate an encrypted signature.
formatRequestUrl() - url.format & url.parse.Process to grab environment variables.setTimeout and clearTimeout as default fallbacks.Buffer to help generate the client secret.
formatRequestUrl() and computeSignature().crypto in conjunction with Buffer to create an encoded signature.encodeURIComponent to encode the signature.google-maps-services-js/lib/internal/make-url-request.js
https uses to make a request.
axios should come in.https.Agent to set keep-alive to true.Url’s parse to parse the URL’s requestOptions.Buffer in conjunction with https’s chunking ability to grab the response in pieces.https.request is the root call that leads to the eventual body is undefined error.google-maps-services-js/lib/internal/task.js
Process.nextTick to ensure the queued task will run in the next event loop.google-maps-services-js/lib/internal/wait.js
setTimeout() for each task that is executed.clearTimeout() for cancelling tasks.These will all have to be swapped out and replaced with a library like axios or superagent. A lot of the interface code, validation code, and even parts of the client side rate limiting should be reusable and remain untouched if done right.
Using my deep dive notes, I went through each task systematically to make sure that I didn’t miss anything that needed editing. Checklists for the win! You can follow along with my notes below as I explain the rationale behind all of my code edits.
google-maps-services-js/lib/internal/cli.js
Process to parse args.
cli.js and run.js.google-maps-services-js/lib/internal/make-api-call.js
method to the queryOptions to be later used in make-url-request. This is either POST (explicit) or GET (implicit default) based on how they wrote their definitions.response.requestUrl to equal requestUrlConfig as we no longer create a url.
Url used to create the URL to send and to generate an encrypted signature.
formatRequestUrl() - url.format & url.parse.
https.request.
params object that I returned to be later used by makeUrlRequest to create the axios instance.useClientId is true to create a encoded signature that is appended onto the end of your query string.
lib/apis files to have baseURL and relativeURL params. This made it so I could use encodeURIComponent to manually build the path + query string needed to do the encode.url.format followed the WHATWG URL spec which looks to do quite a bit more than just encode. You can confirm this by seeing the extra cases that were discussed in this bug report: #1802. Then compare to how encodeURIComponent works: [1] [2] [3] [4]fetch because my experience with it in React Native has shown it to be very lacking. For example: #256 and that abort/cancel is still not finished (something needed by this codebase).Process to grab environment variables.
process.env.NODE_ENV which is shimmed in for compatibility purposes.setTimeout and clearTimeout as default fallbacks.
Buffer to help generate the client secret.
formatRequestUrl() and computeSignature().
buffer is the best way to handle binary data in JavaScript. React Native appears to have a suboptimal implementation for anything related to this. See #10 in 10 Things you should know about React Native.crypto in conjunction with Buffer to create an encoded signature.
toString() on the payload and secret for some reason as I was running into this bug: #32encodeURIComponent to encode the signature.
google-maps-services-js/lib/internal/make-url-request.js
https uses to make a request.
axios should come in.lib/apis files and the return from formatRequestUrl.axios.create in order to create an instance that will later send the request out.Buffer point below.axios.cancel with a generic cancellation message.https.Agent to set keep-alive to true.
Connection to keep-alive which possibly does the same thing. However…httpAgent and httpsAgent params that take the very http/https object that I removed. So this probably does not work as I expect.Url’s parse to parse the URL’s requestOptions.
axios.create() should build an instance with the correct default headers (and other params). So manually creating those params via Url’s parse shouldn’t be needed.url.parse works at the low level, so my assumption above could be incorrect. A bug with the headers could have been introduced here. I may also need to use another library to have a keep-alive agent as suggested by the answers here.Buffer in conjunction with https’s chunking ability to grab the response in pieces.
https.request is the root call that leads to the eventual body is undefined error.
body is undefined in the http response due to the lack of streaming support). This is “fixed” by replacing https/http with axios.google-maps-services-js/lib/internal/task.js
Process.nextTick to ensure the queued task will run in the next event loop.
setImmediate instead on React Native.google-maps-services-js/lib/internal/wait.js
setTimeout() for each task that is executed.clearTimeout() for cancelling tasks.
Showing 15 changed files with 92 additions and 160 deletions.
Explicit Dependencies Added:
Writing up some test code and putting the API through its paces in React Native showed that everything worked! Except geolocate… why?
It turns out that geolocate is no longer functional due to these outstanding issues/PR in axios: #723 #1342 #1121. Tracing the code all the way back, it looks like this bug was introduced in a combination of this commit and this commit that eventually changed the merge order of precedence like so:
So it was a consequence of not thinking through the history of changes and how axios recommends editing configs. The PR should fix this by changing the order to this:
For now, all I can do is wait to bump the axios version one the fix is merged and released.
Running the tests showed that some specs are failing. Meanwhile, all of Google’s spec tests pass.
My Spec Test Results
|
|
So why is this the case? My initial guess was that this had to do with using setImmediate as the tests had to do with timing and task.js. A quick edit of the original source to setImmediate and a run later showed this to be partially the case:
|
|
So for comparison, that’s this difference which doesn’t align perfectly:
|
|
This is an oddity that is going to require a lot more investigation.
For the e2e tests, you’ll find that Google has not updated this library in a long time as the majority of their tests fail. When compared to my fork, you’ll actually get the same errors (+1 for removing the deprecated method):
|
|
I’m not sure how I’ll be tackling these errors as I have no passing baseline to compare against.
Overall, I would say that while work still needs to tbe done, the code is at least usable for my current purposes. If you would like to use this, be aware of the work I’ll be tackling in Future Tasks.
Now to make this fork usable by others without clashing with the original, I had to rename it on Github:
package.json with the correct new information.Then publish it on npm with the new name:
And voila! You can now install and use this package with:
|
|
But a disclaimer: There are lots of outstanding issues that I need to fix before tagging this with a production ready release. So use this at your own risk!!
Forking an publishing projects under a new name isn’t all that simple though. There’s a lot of legal stuff and etiquette that I had to read up on first before I was sure that I should/can do this. IANAL but I’m pretty sure that the Apache 2.0 license that Google uses is ok with this given a bunch of caveats that I have to follow: [1] [2]
Also, remember that you’re building on the shoulders of giants when you fork and use open source work. Do your best to give back, but also be free to build on top of prior with with proper accreditation of course! [1] [2]
You can check the README for a list of immediate tasks that I’ll be tackling. However, in addition to those, I am also considering the following:
placesRadar support so it can at least be used until it’s fully turned off.makeUrlRequest.There’s more that can also be done like using it with React, Angular, etc. I’ve also run into some interesting articles on what you can do when implementing the Google Maps APIs. They’re good source for looking at features can be added after exposing the APIs themselves:
However, all of these extras are outside of my skillset and the current scope of the project. They’re all fun things to think about and hopefully I can tackle them in the future. On that note, I’ll end this post by saying that my first foray into open source has been super fun!
Copyright © 2011-2020 Ho Yin Cheng