Affichage des articles dont le libellé est nodejs. Afficher tous les articles
Affichage des articles dont le libellé est nodejs. Afficher tous les articles
2/26/2020

Validate an openapi or swagger API definition from a Gitlab-CI test step

Lets say you've built $BUILD_IMAGE container image at the build step. I did it on a NodeJS based project but it will work with other technology as well.

check-openapi-contract:
  stage: test
  retry: 1
  timeout: 15m
  script:
    - docker run --name=my-container -d -i -p 8080:8080 --rm $BUILD_IMAGE npm start
    - bash -c 'while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' localhost:8080/swagger.json)" != "200" ]]; do sleep 5; done'
    - docker exec -i my-container curl http://localhost:8080/swagger.json -o ./swagger.json
    - docker exec -i my-container npx swagger-cli validate ./swagger.json

So what do we do? We start the server, then retrieve the swagger.json or openapi.json and leverage swagger-cli validate command to ensure our definition is valid and be notified if it is not. Nothing. More.

1/30/2017

How we reduced by 37% our NodeJS project build time with one line

For Image-Charts, Redsmin and the-to-be-announced-next-SaaS I use Jenkins as the continuous integration system. Lately I discovered that 75% of Image-Charts project build time was related to the dependencies installation part.
I don't like and do not advise to enable caching at the build level because it inherently breaks the principle of repeatable and independent builds thus for every project I work on each build starts with a clean empty cache.

So I wondered how long Image-Charts build would take if we were to replace npm with yarn for the dependencies installation part.



For a medium-sized project like Image-Charts (39 dependencies and 9 dev-dependencies and some native dependencies) switching the installation step to yarn reduced the build time by up to 37% which is really awesome! The next pain point to improve would be a switch from mocha to ava in order to leverage parallel tests execution but that's another story!

4/11/2015

PM2 tips - uncommon ssh port, cluster_mode and deploy troubleshooting

Note: PM2 own code quality is bad and cluster_mode is not production ready. I would not recommend to use PM2, anywhere. You have been warned.

How to use a different ssh port for PM2

Since pm2-deploy directly forward the host string to ssh, you can simply use:

"host": "HOSTNAME -p PORT"

inside the configuration file.

How to use cluster_mode in PM2

Make your app instances listen on port 8000 and pm2 will load-balance any request on 8080 to your app instances.

Troubleshooting pm2 deploy

pm2-deploy outputs its log into /tmp/pm2-deploy.log so if you don't understand why pm2 deploy is not working, a tail -f /tmp/pm2-deploy.log might help.

3/08/2015

[Book Review] MEAN Web Development

After my review of Getting Started With Grunt and Lodash essentials this time Packt Publishing contacted me to review MEAN Web Development. Since Redsmin was first built over the ME(A)N stack and a lot of our applications at Bringr are built using NodeJS/AngularJS I happily accepted the offer, below is the review I posted on amazon.


Chapter 1 Introduction to MEAN


This chapter begins with a brief introduction to the MEAN stack and is then a deep step by step guide on how to install and run MongoDB, NodeJS, NPM both on Windows, Mac OS X and Linux. It's the perfect starting point for newcomers. I will simply add that it's a better practice ? to install NodeJS using Node Version Manager (nvm) that with the official installation software.

Chapter 2 - Getting Started with NodeJS


After a brief history of why Ryan Dahl created NodeJS the author dives into explaining event-driven programming. You will learn how work an event loop and how a webserver with a non-blocking event-loop (nginx) differs from a blocking web-server (apache) in terms of concurrent access performance and memory consumption. Then you'll take a look at closures and why they are useful in callbacks. Finally you'll learn how to write your first NodeJS http server along with your first connect middleware.

Chapter 3 - Building an Express Web App

Time to go to the next step : writing your first Express application, managing sessions, using a template engine (EJS) and a routing scheme. This chapter takes the reader by the hand and is really descriptive on how to organize your application folder architecture and write your first application.
However instead of using configuration files by environments (e.g. production, staging and development) I would recommend the reader to use environment variables (using a module like common-env + autoenv) since it's a way more flexible approach. Another missing point is to warn the reader to add a "private:true" inside its package.json file otherwise it could publish its application on npm by mistake.

Chapter 4 - Introduction to Mongodb


Even if this chapter explains to new comers what MongoDB is, the author is clearly using too much superlative about MongoDB. It's important to recall that MongoDB is not a replacement for relational storage and that it must be used with care.

Chapter 5 - Introduction to Mongoose


Just like the previous chapter, this one is also well written. It explains each Mongoose features starting with schema, validation, virtuals, getter/setter and even DbRef. I would simply note that some conditionals in the code examples could be greatly simplified.

Chapter 6 - Managing User Auth Using Passport


Learn how to handle user creation/connection with PassportJS, using local auth or Facebook/Twitter/Google OAuth. The OAuth user creation mechanism described in the book does not allow a user to connect via multiple OAuth provider and is thus limited. The access_token refreshing mechanism, even if it's not implemented, should be at least given as an exercise for the reader.

Chapter 7 - Introduction AngularJS


Another really well written chapter, this time about AngularJS. From the history to the state of the art, you'll learn everything you need to know to build your first application with AngularJS from routing to services. Note that the authentication service code example does not respect the Inversion Of Control principle and should be rewritten to ask for $window.

Chapter 8 - Creating a MEAN CRUD Module


In this chapter, learn how to set CRUD (Create-Retrieve-Update-Delete) modules up from back (using mongoose) to front (using angularjs $ressource). The middleware approach to retrieve an article is subject to race-conditions. I will just add that a much more powerful alternative than ngressource is Restangular. Also I would advise the reader to prefer ui-router (built over a finite-state-machine with support for sub-views) over angular own router module.

Chapter 9 - Adding Real-time func. Using Socket.io


Next, learn how to communicate in real-time between the browser and the server using websocket. In order to do that you'll learn how to set socket.io up. Be careful about the socket.io session configuration example, the error management should really be improved.

Chapter 10 - Testing MEAN apps


Nice overview of what unit-testing and end-to-end (E2E) testing is about. You'll learn how to unit-test a MEAN application with Mocha and Karma using Jasmine as well as E2E testing with protactor.

Chapter 11 - Automating & Debugging MEAN Apps


The last chapter demonstrates how to configure grunt to automate code quality checking and testing. You will learn grunt, node-inspector (for nodejs debugging) and batarang (a browser extension for angularjs debugging).

Conclusion

MEAN Web Web Development is clearly a well written book that will be the perfect match for beginners in AngularJS and NodeJS!
1/19/2015

Request-api - NodeJS request library as HTTP API

Yep. request-api.
10/19/2014

Check-build - Verify that your NodeJS project follow conventions, is well written and secure

Each time I start a new project/mvp/poc/module I don't want to create/edit a new make/grunt/gulp file or whatever hype dev use these days. I want an already packed CLI with good defaults (mine) that I can drop into my continuous build/integration process. Let's build that once and for all.
– 10/19/2014

check-build leverage jshint, jscs, jsinspect, buddyjs and nsp, to enforce DRYness, coherent coding-style, prevent errors and automatically check security issues.



check-build will be gradually integrated into all our projects at Bringr, Redsmin, UserAPI and the soon to be announced Spid project.

8/25/2014

Dot-clipboard - monitors your clipboard and runs scripts based on its content

When I first thought of — and implemented — clipboard-watcher I knew I will then have to build something like dot-clipboard.

It tooks me a few hours of hard passionate work with NodeJS fs.watch and clipboard-watcher to build the core and a few more hours to write the first download-gif script.

I really liked playing with the idea of script hot-reload and auto-checking. Actually some of these parts are already re-used inside an upcoming project that is targeting a much larger problem : website and SaaS performance... At the time of writing we already have a functional prototype but I will share more on this later...

Coming back to dot-clipboard, the release went well : first page on Hacker News, more than 3 000 unique visitors on Github and some interesting tweets :




Now let's see what scripts the JavaScript/NodeJS community will build!

5/05/2012

How to fix the "Could not decode a text frame as UTF-8." bug

Sometimes Google Chrome throw a Could not decode a text frame as UTF-8 error. It happens when the server send invalid unicode characters (see Unicode surrogates) to the browser (via websockets or any other transport) and . I've found two work-around for this issue.

The first one is from my point of view, the best approach (the original code came from SockJS codebase). It removes all the invalid unicode characters from the string so you can send it from the server-side without further decoding.

/*
 * Fix the "Could not decode a text frame as UTF-8." bug #socket.io #nodejs #websocket
 *
 * Usage:
 *   cleanedString = filterUnicode(maybeHarmfulString);
 *
 * Original work-around from SockJS: https://github.com/sockjs/sockjs-node/commit/e0e7113f0f8bd8e5fea25e1eb2a8b1fe1413da2c
 * Other work-around: https://gist.github.com/2024272
 * 
 */

var escapable = /[\x00-\x1f\ud800-\udfff\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufff0-\uffff]/g;

function filterUnicode(quoted){

  escapable.lastIndex = 0;
  if( !escapable.test(quoted)) return quoted;

  return quoted.replace( escapable, function(a){
    return '';
  });
}

The second one takes another approach which seems valid (I only tested the former) but requires an extra decoding step on the other side:

/**
 * encode to handle invalid UTF
 * 
 * If Chrome tells you "Could not decode a text frame as UTF-8" when you try sending
 * data from nodejs, try using these functions to encode/decode your JSON objects.
 * 
 * see discussion here: http://code.google.com/p/v8/issues/detail?id=761#c8
 * see also, for browsers that don't have native JSON: https://github.com/douglascrockford/JSON-js
 * 
 * Any time you need to send data between client and server (or vice versa), encode before sending,
 * and decode upon receiving. This is useful, for example, if you are using socket.io for real-time
 * client/server communication of data fetched from a third-party service like Twitter, which might
 * contain Emoji, or other UTF characters outside the BMP.
 */
function strencode( data ) {
  return unescape( encodeURIComponent( JSON.stringify( data ) ) );
}

function strdecode( data ) {
  return JSON.parse( decodeURIComponent( escape ( data ) ) );
}

Hope this help !

[Update] Dougal Campbell made some important notes: “the second method preserves the original data, while the first strips out information, altering the original data”. Thus, the first method can lead to potential security leaks (see his comment).

3/06/2012

NodeJS process management at Brin.gr

I saw today a question on StackOverflow about "Running and managing nodejs applications on a single server" and thought it would be a good idea to share how we deal with NodeJS applications at brin.gr.

  • First we use supervisord to manage & automatically restart applications. The configuration file for each application looks like this:




  • Finally, for remote control and application monitoring, we've setup Monit. Each application has a monit configuration file like the following:


Note: before this current workflow we were using forever/forever webui but forever remained quite unstable so we decided to switch to supervisord/monit and since this migration everything's fine.
12/15/2011

Forever-WebUI, Node-AMQP-dsl & Node-AMQP-tool

I released three library/tool/cli this last months: Forever-WebUI, Node-amqp-dslNode-amqp-tool.

Let's start with Forever-WebUI. I use forever at Brin.gr for managing node processes but I've always found the workflow slow when wanting to restart multiple scripts. So I developed a web interface that lists all running scripts and allow the user to browse logs, restart or stop scripts.

npm install forever-webui && node node_modules/forever-webui/app.js
Note: if someone here knows how to update a backbone collection's view without having to remove the entire list and adding again one by one each view feel free to enlight me ;)

Next Node-AMQP-dslNode-AMQP-tool: Brin.gr rely heavily on RabbitMQ and as usual when using the same tool everyday the little annoying things quickly become a nightmare. So AMQP-dsl is a fluent interface for node-AMQP (more examples on Github) ...


And AMQP-tool is a simple CLI to easily import & export AMQP queue. For example, exporting 5000 messages from a queue into a file is easy as:

amqp-tool --host rabbitmq.local -u user -p azerty -q queuetest --count 5000 --export > dump.json
7/18/2011

[Week-end Project] Nodejs language detection library using n-gram

Node-language-detect is a NodeJS port of the PEAR package Text_LanguageDetect by Nicholas Pisarro.

2/25/2011

Growl Mac pour l'API de notification Google Chrome (poc)

Extension Google Chrome Growl pour les notifications web
Peut-être avez-vous vu passé ce tweet:
Je comprends tout à fait que le staff en charge de Chrome ne souhaite pas développer des extensions spécifiquement pour Mac (ou Windows ou Linux). Mais même si Growl n'est pas intégré en natif sous Mac, il est dommage de ne pas en profiter lorsqu'il est présent sur la machine. L'autre raison de cette non implémentation est que Growl ne gère pas les notifications de type HTML alors que ce type de notification est possible via l'API Web Notification.

Enfin si aucune extension Google Chrome pour Growl Mac n'existe c'est parce qu'il est impossible d'appeler un exécutable (dans notre cas Growlnotify) depuis une extension (pour des raisons de sécurité bien évidement).

Il faudrait donc pouvoir exécuter un exécutable depuis une extension Google Chrome. Une solution possible est de communiquer via websocket avec un serveur NodeJS local qui se chargera d'exécuter Growlnotify:


L'extension ChromeGrowl surcharge la méthode createNotification ainsi que createHTMLNotification de l'API de notification. Voici l'extension en action:


Le support de createHTMLNotification est pour le moment quasiment inexistant car dans la majorité des cas une requête cross-domain est requise. La solution serait d'envoyer l'url au serveur node qui téléchargerait le contenu puis après analyse afficherait le plus d'informations possible via Growl.

Le code est disponible sur mon GitHub Chrome-Growl-Notification. Il s'agit d'un proof of concept réalisé en quelques heures n'hésitez pas à le forker pour l'améliorer ou corriger de possibles bugs.

EDIT: Je viens de découvrir qu'il faut payer une taxe de 5$ pour pouvoir publier des plugins sur le Google Chrome Extensions Directory. Je ne compte pas verser le moindre $ pour un petit POC, néanmoins si l'envie vous en dit vous pouvez toujours faire un don :).

Sources:
»
 
 
Made with on a hot august night from an airplane the 19th of March 2017.