Querying the Web with Node.js



Resources
Principle

Querying the Web is a common repeated task with Node.js. The use of native http and https native libraries may become tedious; third-party libraries like Axios may do the job in a more seamless way.

Web scraping is a more advanced usage of Web querying with dedicated libraries like Puppeteer.

Querying the Web ⤳ https (native) module

import * as https from 'https';

https.get("https://www.carrefour.fr/s?q=3083681093926", (response) => { // Aubergines Cassegrain
    let data = '';
    response.on('data', chunk => { // A chunk of data has been received...
        console.info(data);
        data += chunk;
    });
    response.on('end', () => { // The whole response has been received...
        try {
            console.info("\nThe end...");
        } catch (exception) {
            console.log(`${exception.name}: ${exception.message}`);
        }
    });
}).on('error', (error) => {
    console.log("Error: " + error.message);
});
REpresentational State Transfer -RESTful- Web services here… and there…

Express is a direct support for programming REpresentational State Transfer -RESTful- Web services.

Express.json() here… allows the parsing of JSON data for POST and PUT requests. 

const PLM = Express();
PLM.use(Express.json()); // For POST (and PUT) requests (based on 'body-parser' library in Node.js)...
const port = process.env['PORT'] || 1963; // 'process' => "@types/node"
const server = PLM.listen(port);
console.log(`Server ready to accept requests on port ${port}... ctrl + C to abort...`);
GET requests
PLM.get('/', (request, response) => { // Test: 'curl http://localhost:1963/'
    response.send('
"PLM.Node.js.ts": Restful Web services
');
});
PLM.get('/api/PLM/Articles', (request, response) => { // Test: 'curl http://localhost:1963/api/PLM/Articles'
    response.send(Articles);
});
PLM.get('/api/PLM/Articles/:reference', (request, response) => { // Test: 'curl http://localhost:1963/api/PLM/Articles/CD100'
    const article = Articles.find((a: Article) => a.reference === request.params.reference);
    if (!article) response.status(404).send('<h1 style="color: red;">Not found: ' + request.params.reference + '</h1>');
    response.send(article);
});

Reminder on HTTP status codes here…

POST requests
// Test: curl -X POST http://localhost:1963/api/PLM/Nouvel_article -H 'Content-Type: application/json' --data '{"reference":"CC201","designation":"Camion citerne rouge","type_fabrication_achat":"Fab. a la commande","unite_achat_stock":"unite","delai_en_semaine":2,"lot_de_reapprovisionnement":150,"stock_maxi":600,"PF_ou_MP_ou_Piece_ou_SE":"PF"}'
PLM.post("/api/PLM/Nouvel_article", (request, response) => {
    /* Création article (Windows 11) :
    $CC201 = @{
        reference = 'CC201'
        designation = 'Camion citerne rouge'
        type_fabrication_achat = 'Fab. a la commande'
        unite_achat_stock = 'unite'
        delai_en_semaine = 2
        lot_de_reapprovisionnement = 150
        stock_maxi = 600
        PF_ou_MP_ou_Piece_ou_SE = 'PF'
    }|ConvertTo-Json
    echo $CC201
    */
    /* (Windows 11) :
    $Nouvel_article = @{
        Body        = $CC201
        ContentType = 'application/json'
        Method      = 'POST'
        Uri         = 'http://localhost:1963/api/PLM/Nouvel_article'
    }
    */
    /* (Windows 11) :
    Invoke-RestMethod @Nouvel_article
    */
    // console.log(JSON.stringify(request.body));
    const article: Article = request.body;
    const {error, value} = Article_.validate({
        reference: article.reference, // Unicity required as well...
        designation: article.designation, // Unicity required as well...
        type_fabrication_achat: article.type_fabrication_achat,
        unite_achat_stock: article.unite_achat_stock,
        delai_en_semaine: article.delai_en_semaine,
        lot_de_reapprovisionnement: article.lot_de_reapprovisionnement,
        stock_maxi: article.stock_maxi,
        PF_ou_MP_ou_Piece_ou_SE: article.PF_ou_MP_ou_Piece_ou_SE
    });
    if (error === undefined) {
        response.status(201).json({
            message: "Article validé (succès) : " + value.reference
        });
        Articles.push(article);
    } else {
        response.status(400).json({
            message: "Article invalidé (échec) : " + error.message
        });
    }
Data validation with joi

The importance of data validation within the middleware tiers requires devoted libraries like joi or express-validator that supports data validation within Express. 

import * as Joi from 'joi';

export enum Article_type {
    MP = 'MP',
    PF = 'PF',
    Piece = 'Piece',
    SE = 'SE'
}

export interface Article {
    reference: string;
    designation: string;
    type_fabrication_achat: string;
    unite_achat_stock: string;
    delai_en_semaine: number;
    prix_standard?: number;
    lot_de_reapprovisionnement?: number;
    stock_mini?: number;
    stock_maxi?: number;
    pourcentage_de_perte?: number;
    inventaire?: number;
    PF_ou_MP_ou_Piece_ou_SE: Article_type
}

export const Article_ = Joi.object({
    reference: Joi.string().alphanum().required().min(4).max(10),
    designation: Joi.string().pattern(new RegExp('^[a-zA-Z0-9 ]*$')).required().max(30),
    type_fabrication_achat: Joi.string().pattern(new RegExp('^[a-zA-Z0-9. ]*$')).required().max(30),
    unite_achat_stock: Joi.string().pattern(new RegExp('^[a-zA-Z0-9 ]*$')).required().max(30),
    delai_en_semaine: Joi.number().integer().required(),
    prix_standard: Joi.number(),
    lot_de_reapprovisionnement: Joi.number().integer(),
    stock_mini: Joi.number().integer(),
    stock_maxi: Joi.number().integer(),
    pourcentage_de_perte: Joi.number(),
    inventaire: Joi.number().integer(),
    PF_ou_MP_ou_Piece_ou_SE: Joi.string().valid(Article_type.MP, Article_type.PF, Article_type.Piece, Article_type.SE)
}); // .id('Article_'); // 'id' is for possible *INTERNAL* reuse, i.e., '#Article_'
Data validation with joi ⤳ uniqueness, i.e., SQL Unique or Primary key 
Article_.external(async (article: Article) => { // Check uniqueness in database
    console.log("Access to database from primary key: " + article.reference);
    // Code here...
}).validateAsync({
    reference: 'CD100',
    designation: 'camion demenagement bleu',
    type_fabrication_achat: 'fab. par lot',
    unite_achat_stock: 'unite',
    delai_en_semaine: 2,
    lot_de_reapprovisionnement: 200,
    stock_maxi: 600,
    PF_ou_MP_ou_Piece_ou_SE: Article_type.PF
});
Data validation with joi ⤳ nesting 
interface Lien_de_nomenclature { // 'compose !== composant'
    compose: Article;
    composant: Article;
    quantite_de_composition: number;
}

export const Lien_de_nomenclature_ = Joi.object({
    // 'exist()' <=> 'required()'...
    compose: Article_.exist(),
    // 'compose' et 'composant' ont des valeurs distinctes pour 'reference' :
    composant: Article_.exist().when('compose.reference', {
        // is: Joi.exist(), // Useless, already checked...
        then: Joi.object({reference: Joi.not(Joi.ref('...compose.reference'))})
    }),
    quantite_de_composition: Joi.number().required()
});

const validation = Lien_de_nomenclature_.validate({
    compose: {
        reference: 'CD100',
        designation: 'camion demenagement bleu',
        type_fabrication_achat: 'fab. par lot',
        unite_achat_stock: 'unite',
        delai_en_semaine: 2,
        lot_de_reapprovisionnement: 200,
        stock_maxi: 600,
        PF_ou_MP_ou_Piece_ou_SE: Article_type.PF
    },
    composant: {
        reference: 'P005', // Replace by 'CD100' for test...
        designation: 'phare normal',
        type_fabrication_achat: 'achat par lot',
        unite_achat_stock: 'unite',
        delai_en_semaine: 2,
        prix_standard: 0.75,
        lot_de_reapprovisionnement: 500,
        stock_mini: 750,
        stock_maxi: 1500,
        PF_ou_MP_ou_Piece_ou_SE: Article_type.Piece
    },
    quantite_de_composition: 2
});
if (validation.error === undefined)
    console.log("Succès : " + validation.value.compose.reference + "-" + validation.value.composant.reference);
else
    console.log("Echec : " + validation.error.message);

Exercise