Appendix A tech professionals (developers, designers and ... · NOVA – School Of Business And...

NOVA – School Of Business And Economics Retargeting – the case of Landing.jobs

Miguel Pereira, 2592 1

Appendix A – tech professionals (developers, designers and data scientists)

A software developer is someone whose job is centred around the creation of software - these

individuals are also called programmers. Usually, a tech professional knows more than one

programming/markup language, and depending on the type of programming languages and

software those professionals master, they can be classified as web developers (front-end and

back-end developers), data scientists, designers and/or mobile developers.

Front-end developers and designers

Front-end developers work on the so-called ‘client side’ actions – the components of a website

that users will actually see and experience when they use the site or app. “Front-end developers

might work on: the functions of buttons, the layout of pages, the menus and structures of an

application or website, and the organization and user experience within a shopping cart.” (What

The Dev, 2015). A front-end developer should know how to code in at least one of the following

languages: HTML, CSS, jQuery, Javascript, Hive, Postgres, Pig, XML-based languages,

XHTML, HTML5, Angular.js, Node.js, Cofeescript. More recently, some companies expect

front-end developers to have designing skills – since this type of developers work on the visible

site of the digital product, makes sense that apart from the coding skills they also hone image

and video editing skills. In this cases, web designers must know how to work with specific

designing toolkits, like Adobe Illustrator and Adobe Photoshop.

Back-end developers and Data scientists

Back-end developers work on the ‘server side’, to ensure everything runs properly – this type

of developers aren’t concerned with a website’s and/or application appearance and aesthetics,

only that it performs perfectly. “Here are things that a back-end developer might be responsible

for: ensuring that the customer data is saved and organized correctly; keeping a website secure;



having shipping costs calculate automatically in your shopping cart; scraping search engines

for data; sending out emails automatically from your server.” (Zephoria, 2015). A back-end

developer should know how to code in at least one of the following languages: Perl, C, C++,

SAS, R, Python, SQL, Visual Basic, MySQL, Matlab programming, PHP, Java, Ruby, SQL,

Candle, ASP.Net. Data scientists on their turn, are “part mathematician, part computer scientist

and part trend-spotter” (SAS, 2016; Udacity, 2014). A data scientist’s job is to dive deep into

raw data and analyse it, so that business insights can be withdrawn from its analysis. In order

to do this, the data scientist should have a solid background in mathematics and algorithms, as

well as a good understanding of human behaviours and the nature of the markets and industries

the analysed dataset affect. These tech professionals ideally should be familiarized and know

how to code in languages such as SAS, R or Python, manage their way around with databases

and data visualization and reporting technologies (SAS, 2016).

Mobile developers

This is the type of developers who develop, maintain and optimize mobile applications. In the

digital technology space, mobile refers to “pertaining to or noting a cell phone, usually one with

computing ability, or a portable, wireless computing device used while held in the hand, as in

mobile tablet; mobile PDA; mobile app.” (Dictionary, 2016). Mobile developers can therefore

be of several sub-types, depending on a specific job or task they are assigned to, which means

that they can either be, at the same time, front-end developers and/or back-end developers.

However, usually they are expected to master specific programming languages, such as Swift

or Objective-C (to develop mobile applications for iOS apps).

http://blog.udacity.com/2014/11/data-science-job-skills.html



Appendix B – LJ’s job curation process and product range

Both the process and product range are from the author of this study, based on Landing.jobs

(2016).

The process

The company has a three-stage process to filter down candidates applying for one of its

client’s job opportunities:

1) Attract – in this stage, LJ focus its attention in raising brand awareness, via employer

branding, content marketing and referral rewards (people who refer a developer are

rewarded a certain amount of cash, paid by the company who launched the job opportunity,

if the person recommended effectively gets the job). LJ also does not accept every

organisation on its platform – all companies that would like to be featured go through a

curation process and only the ones who meet LJ’ requirements enter the company’s client

pool;

2) Evaluate – in this stage, through their own technology and external advisors, the company

uses scoring and smart filters, as well as specific tech challenges for the position at stake

and also making use of its tech curators network to narrow down the potential hiring list;

3) Engage – in the last stage, the company helps its clients who successively hired a candidate

to manage their new employees, providing on-going assistance through smart notifications,

career advisory, in-app messaging and also an applicant tracking system for candidates.

Product range

The company currently offers two products to its customer base:

1) Landing.applications (pay per post) – includes job offers posting and curation, matching

and scoring algorithms, smart filters, smart notifications to candidates and access to LJ’

external tech curators network. In order to purchase this product, clients are required a



minimum of 150 €, that includes five vetted applications, being able to pay 30 € per

additional vetted application, charged on a monthly basis. According to the company, on

average, 20 applications turn into a hire;

2) Landing.hires (pay per hire) – includes all of the features of the Landing.applications

product and also an account manager, proactive sourcing, pre-screening calls, tech testing

and candidate engagement. LJ charges 11% of the Gross Annual Sallary (GAS) of the

hired candidate or 9% of the GAS if the client pays an upfront retainer of 500 €. LJ also

offers a guarantee in which if the candidate leaves within 90 days from the starting date, a

replacement will be found, free of charge.

Appendix C – How to set up a Facebook campaign

In order to set up a Facebook campaign, the platform demands the advertiser to follow a three-

layer particular structure:

1) Campaign – It is the top layer of the campaign and contains one or more advert sets and

adverts. Every campaign holds one and one only main advertising objective for each

campaign;

2) Advert set – The second layer of the campaign that contains one or more adverts. On

this layer the advertiser can define its targets, budget, schedule, bidding and placement;

3) Advert – The final layer which represents the ad that is actually going to be displayed

and this is where the advertiser can upload an image or video, include call-to-actions,

links and text.



Appendix D – Facebook audiences

Types of audiences

a) Fans and followers

Every Facebook user who “likes” a Facebook page is called a fan. However, not all fans are

followers of the page. Once someone “likes” a Facebook page, the user automatically becomes

a follower of the page too. A follower is a user who is going to receive notifications from the

page every time there is an update. It is even possible to set a preference on a particular page,

by manually selecting the “See first” option on the newsfeed section – when selecting this

option, every time the page posts something this post will appear on the top of the user’s

newsfeed, thus being in a highlighted position. However, users can “like” a page and not follow

it at the same time – they display interest on the page but do not want to be constantly notified

on the most recent activities. This is quite an important distinction, since the engagement

between a Facebook page and a follower is usually much higher than the one of a user who

only “likes” the page, for obvious reasons (WeSellLikes, 2016).

b) Custom audiences

According to (Facebook, 2016c), custom audiences represent a very effective and efficient way

to target Facebook ads. Through this type of audience, it is possible to create a very high niche

user base from several types of user data, such as lists of email addresses, phone numbers,

website visitors, fan page members and even engagement. There are four types of custom

audiences: standard custom audience, website custom audience, app activity custom audience

and lookalike audience.

1) Standard custom audience

Audience created based on a list of emails, phone numbers, or Facebook User Ids. Then,

Facebook is going to match those users with its own users, which according to Adspresso this



matching is usually of 60-80% (that is, from all of the users’ contacts imported from the

company’s data warehouse, 60-80% will be accurately match with current Facebook users).

2) Website custom audiences

Website Custom Audiences can be built using the Facebook Pixel on the company’s website.

Using the pixel, it is possible made of every user that has visited a specific page the company’s

website during a set time period of up to 6 months (180 days). Essentially, this is a retargeting

custom audience: a user base of people who have already visited your website and that are still

relevant to the company, there is a chance that by reengaging with these users new conversions

may arise.

3) App activity custom audiences

This is a type of audience made of users who have interacted with a company’s app in some

way (simply opened it, made a purchase, filled out a form, among many other types of

interactions).

4) Lookalike audiences

According to (Facebook Business, 2016), lookalike audiences are a type of audience of people

who have a similar online navigation and action behaviour to your current audiences. Facebook

will look for patterns and characteristics the company’s current users have in common (such as

demographics and interests) and generate from that a much larger group of identical users. This

type of audiences can be created based from several distinct types of user data, like the

company’s Facebook page fan base and frequent website visitors.

https://www.facebook.com/business/help/231114077092092



Appendix E – interview guide and answers (summary of both interviews)

TABLE 1 - interview guide and answers (summary of both interviews)

Topics Key insights

1) Retargeting in the case of LJ Yes, it does make sense. Setting up a strategy of this kind for a digital company can cut down costs

dramatically, mainly because it automates some key business processes (in their case, prospecting

for new candidates). Moreover, the entire process and respective results can be continuously

optimized and measured, thus allowing the company to be focused on the core aspects of its

business.

2) Resources This is a very subjective question that depends on a lot of different factors, so there is no absolutely

correct answer. Everything boils down to the marketing objectives, available budget and timeframe

to conduct the tests. One should not spend much money in the beginning – would rather make

small investments on a few posts and get some insights and only later, eventually, start increasing

the spend. Considering the amount of constraints they have, one should try to keep things as simple

as possible, not only on the structure of the posts but also on the way these are analysed and



optimised. This makes it is easier for them to understand and allowing them to continue with the

current strategy.

3) Strategy, tactics and best practices The most important metrics one should take into account when devising this type of strategy are

1) amount of unique users per time period (usually, months); 2) CTR, CR, CPM and CPC

(depending on the specificities of the campaign); 3) demographics; 4) traffic sources. Both experts

had never used academic literature (for example, scientific papers) in their work and they were

also unaware of anyone in this field who uses it too. There are a lot of companies that often publish

their own studies, which come from their own experience and daily work with their customers,

with insights on industries, practices, as well as regarding any other relevant topics. The most

reputed ones are Kissmetrics, Adroll, MOZ, Hubspot, among others – there are plenty of highly

reliable sources out there, the amount of information is tremendous. These are the ones they go to

whenever they want to get an update on the latest breakthroughs.

Regarding the strategy itself, the experts suggested to focus on simplicity above everything else.

If LJ’s team does not have someone with the required expertise to effectively put a very complex

digital strategy in place and if they are also not planning in having one in the short/medium-run,



as soon as the author leaves the project all the tests that were ran, as well as the future ones, need

to be as clear as possible for them to understand.

The experts also suggested to start with small tests with a small budget (somewhere between 5-15

€ per post) during two weeks and at the end of those two weeks one should look at the results and

optimize the ads based on those outputs.

In their opinion, this is the best way to go since spending a lot of money on these posts will

probably make the frequency cap go through the roof: the audience is very niche, so if putting a

lot of money into those posts, Facebook will have no alternative than to show the same ad a lot of

times to the same people because there is not more to people to target. Moreover, investing that

much right in the beginning is not an efficient marketing budget allocation if your goal is only to

test some hypotheses. Investing heavily on Facebook makes sense when you want to get actual

results in a short period of time, which is not the case.

I would also not include too many settings in what regards targeting all at once – for example, one

should try instead including gradually different interests to test what works best and what doesn’t.

One should do this because in this way it is easier to test what works best and what does not – by

putting all of them at the same time, it will be harder for you to measure the impact of each variable



on the performance. Additionally, even though one should be as specific as possible in what comes

to targeting, which is one of the techniques that usually leads to the best possible results, due to

the fact that the audience is niche, including too many specificities may generate an insignificant

amount of users.



Appendix F – expert profiles

Expert #1

Expert #1 works for one of the largest retailers in Portugal, as a Conversion Specialist, while

doing research on that retailer’s Innovation Lab simultaneously. The expert contacts on a daily

basis with similar problems and using these sort of platforms is part of the expert’s job. The

expert is also a tutor at EDIT, arguably one of the best Digital Marketing Academies in

Portugal, teaching courses on Digital Strategy and how to use specific digital marketing-related

software toolkits. On top of that, the expert also has relevant international experience in

business-related activities: took an Executive Course at the Harvard Business School; holds a

Bachelor’s degree in Management from one of the best Portuguese business schools; took an

Executive Education program from one of the best Portuguese business schools; worked for

Rocket Internet and co-founded one of the largest ecommerce food delivery businesses in Asia.

Considering both the academic and professional background of the expert, it was believed that

the expert would provide relevant and reliable insights on the topic, thus enriching the author’s

knowledge on the subject, leading to a better informed of the problem.

Expert #2

Expert #2 works as a Senior Performance Marketing Manager at the Lisbon offices of one of

the world’s biggest and most important digital marketing agencies. The expert contacts on a

daily basis with similar problems to the one of LJ and using these sort of platforms is part of

the expert’s job. Prior to that, the expert also worked for a smaller advertising agency, as a

Digital Business Manager, and also two of the world’s largest retailers. The expert holds a

Bachelor’s degree in Marketing from one of the best Portuguese school of communication and

media students, currently seeking a Master’s in Management from one of the best Portuguese

business schools. Considering both the academic and professional background of the expert, it



was believed that the expert would provide relevant and reliable insights on the topic, thus

enriching the author’s knowledge on the subject, leading to a better approach of the problem.

Appendix G – list of abbreviations

The following definitions are from the author of this study, based on Shopify (2016).

CPA – cost per action (investment / total number of actions). An action is often considered a

conversion;

CPM – cost per one thousand impressions (investment / (total number of impressions / 1000));

CPC – cost per click (investment / total number of clicks);

CTR – click through rate (total number of clicks / total number of impressions);

CR – conversion rate (total number of actions / impressions);

Impressions – Designation used for every time an ad is exhibited (printed);

Avg. – abbreviation for the word “average”;

Ad – abbreviation for the word “advertisement”.



Appendix H – previous Facebook posts from LJ

TABLE 2 – EXAMPLE OF AN AD IN WHICH THE RETARGETING TECHNIQUE WAS NOT USE (AVERAGE PERFORMANCE)

Previous ad #1 - representative of the average performance of LJ's posts (that is, on average, LJ's ads displayed similar results to

this one)

Optimisation Performance

Results (conversions) Cost (per conversion) Budget Periodicity Reach Impressions Clicks Avg. CPM Avg. CPC CTR Frequency cap CR

10 1,00 € 10,00 € Daily 1655 2119 32 4,719 € 0,3125 € 1,51% 1,28 0,60%

TABLE 3 – EXAMPLE OF AN AD IN WHICH THE RETARGETING TECHNIQUE WAS NOT USE (LOW PERFORMER)

Previous ad #2 - low performer (in terms of CR)



14 1,07 € 15,00 € Daily 3742 4390 77 3,417 € 0,1948 € 1,75% 1,17 0,37%



TABLE 4 – EXAMPLE OF AN AD IN WHICH THE RETARGETING TECHNIQUE WAS NOT USE (TOP PERFORMER)

Previous ad #3 - top performer (in terms of CR)



31 0,24 € 7,50 € Daily 1522 2289 51 3,277 € 0,1471 € 2,23% 1,50 2,04%



Appendix I - Naming Structure – Facebook posts

The naming of the Facebook posts goes according the following structure: “[type of audience]” + “_” + “[desired programing language(s) for the

job opening]” + “_” + “[audience group]” + “_” + “[number of the ad]”. The “number of the ad” field corresponds to its respective order in the set

of ads – the first ad was CustomW_Java_G_1, the second one CustomW_Java_G_2 and CustomW_Java_G_3 the third and final one. This structure

was the one being used by the company, so in order to comply with LJ’ procedures, it was the one used for the subsequent tests:

TABLE 5 – NAMING STRUCTURE OF LJ’S FACEBOOK POSTS

Naming structure – Facebook posts

Types of

audience

Tag Description

Website custom

audience

CustomW_Java_G

Custom (Custom) audience built up from the user data collected by the Facebook pixel, on people that

may be suited for the job opening, due to his/her expertise in Java (Java) development and that have

visited the Landing.jobs website (W, Website Custom Audience) over the last 30 days (G).



TABLE 6 - CustomW_Java_G_1 (INITIAL SETTINGS)

Sub-period #1 – initial settings

Post

Conversion Budget & schedule Audience Placement Optimisation and delivery

Conversion

event

location

(type)

Daily

budget

Schedule

start

Schedule

end

Custom

audience

Locations Age Gender Languages

Detailed

targeting

Device

types

Optimisation for

advert delivery

Bid

amount

CustomW_Java_G_1

Website or

Messenger

(purchase)

Daily

budget

Nov 1 Nov 14

Visits 30

days No

Conv_1

PT 21-35 M/F

PT, EN1,

EN2

None

Desktop

only

Conversions Automatic



Appendix J - Facebook posts – settings and respective results

CustomW_Java_G_1

Settings (characteristics of each post):

Conversion type: Purchase – actual application to one of the job opportunities.

Daily budget: 7,50 € (estimated reach of 200-500 Facebook users).

Schedule start: Nov 1.

Schedule end: Nov 14.

Custom audiences: Visits 30 days No Conv_1 – everyone who visited https://landing.jobs in

the last 30 days before the ad started to run, but who did not apply to any job opening (initial

set of users).

Locations: PT (PT stands for Portugal). Rationale: The main target was users who are currently

living in Portugal. Although there may be people outside of Portugal that could be interested

and suited to the job openings, since those are likely very few users, including those prospect

candidates would dramatically increase the size of the audience while reducing exponentially

the accuracy of the targeting at the same time. For example: if users who live in Spain were

included in this audience, its size would increase tremendously. However, it is unlikely that

those targeted Spanish users know how to communicate proficiently in Portuguese, are willing

to come to Portugal to work or even that they would be interested in those particular job

openings. Therefore, even though the size of the audience would become larger, the efficiency

and effectiveness of the ad would be much smaller, especially in what concerns cost per

conversion.

Age: 21-35. Rationale: It is hard to believe that anyone at the age of eighteen already has

relevant professional experience in performing similar roles to the ones of the considered job

openings. Moreover, since a bachelor’s degree in computer science (or any equivalent

bachelor’s degree) takes never less than three years, the minimum age set was twenty-one.



However, due to the scarcity of tech professionals who hone skills like Java development,

someone coming straight out of university (assuming it completed its studies by twenty-one)

may already be suited for the job. Thirty-five was set as the maximum age, as usually, for this

type of positions, someone who is older than thirty-five occupies a highly senior role in an

organisation and has a minimum of ten years of experience. This fact makes this type of

professionals extremely expensive for the company and for this particular position it was not

necessary someone with that level of expertise. These insights were a suggestion from LJ’s

team, from their own experience.

Gender: M/F (M stands for male, F stands for female).

Languages: PT, EN1, EN2 (PT stands for Portuguese, EN1 stands for English from the United

Kingdom, EN2 stands for English from the United States of America). Rationale: Even though

the position is for a native Portuguese speaker and also to work on the Lisbon offices of the

client, this setting refers to the language of Facebook page set by the user. A lot of Portuguese

users have English as the selected language for both their browsers and Facebook account,

which means that if the language chosen was only PT, the ad would not be displayed to users

with English as the selected language. Furthermore, some users prefer English from the United

Kingdom over English from the United States as the selected language and vice-versa and that

is why both EN1 and EN2 were included.

Detailed targeting: None. Rationale: For the initial post it was considered only the

characteristics of the custom audience, so other settings could be considered later as a result of

the performance outputs of the post to test new hypotheses.

Device types: Desktop only. Rationale: To submit an application, there is a lot of information

that the applicant must provide, including completing the registration on the platform as part of

the process. Therefore, it is fair to assume that users will not do this on their phones – this

insight was also a suggestion from LJ’ team, from their own experience.



Optimisation for advert delivery: Conversions. Rationale: Facebook will deliver the ad in the

way that maximizes conversions according to its ad delivery algorithms.

Bid amount: Automatic. Rationale: Facebook will automatically adjust the bid amount for

optimal ad delivery (maximum number of clicks for the lowest price possible) via its ad delivery

algorithms.



TABLE 7 - CustomW_Java_G_1 (RESULTS)

Sub-period #1 - Results

Post



CustomW_Java_G_1 28 0,27 € 7,50 € Daily 1474 2060 52 3,41 € 0,1442 € 2,52% 1,40 1,9 %

CustomW_Java_G_2


Sub-period #2 – Interests (new settings)

Post


Conversion

event

location

Daily

budget

Schedule

start

Schedule

end

Custom

audience


Detailed

targeting

Device

types

Optimisation for

advert delivery

Bid

amount

CustomW_Java_G_2

Website or

Messenger

-> Purchase

Daily

budget

Nov 15 Nov 30 Visits 30 days

No Conv_1

PT 21-35 M/F PT, EN1,

EN2

Set of

interests

Desktop

only




All of the previous settings used in CustomW_Java_G_1 (conversion type, daily budget,

schedule start, schedule end, languages, device types, bid amount, age and optimisation for ad

delivery) were maintained except detailed targeting and custom audience. The new custom

audience was “Visits 30 days No Conv_2” - everyone who visited https://landing.jobs in the

last 30 days but who did not apply to any job opening (counting from the first day the ads using

the custom audience Visits 30 days No Conv_1 started running). In order to increase

granularity and cut down the conversion costs, a set of relevant interests was included in the

targeting section, narrowing down the total size of the target audience – the new target audience

is made of everyone that had already been seen the ad in the previous period and that also was

interested in the set of interests included.

Detailed targeting: Set of interests: “Java”; “Java development”; “Back-end development”.

Rationale: “Java” alone is a bad interest to use, since the word is also the name of an island in

Indonesia. Besides this, Java is one of the most famous programming languages and there are

a lot of people who follow Java-related pages due to its popularity, even if they did not have

any previous contact with coding in this language (or with any programming language at all).

Therefore, some similar terms were included, the ones mentioned. Users who have liked pages

or interacted with ads that are related in some way with those subjects will narrow down the

initial audience, making it more specific, which may lead to a decrease in the conversion costs.

Consequently, the target audience will be made of people who visited https://landing.jobs in

the last 30 days before the ad started to run, did not apply to any job opening (initial set of

users) and also displayed an interest Facebook pages and ads who are in some way related with

the topics “Java”, “Java development” and “Back-end development”.




Sub-period #2 – Interests (results)

Post


Results (conversions) Cost (per conversion) Budget Periodicity Reach Impressions Clicks Avg. CPM Avg. CPC CTR (%) Frequency cap CR (%)

CustomW_Java_G_2 19 0,39 € 7,50 € Daily 974 1658 44 4,524 € 0,1705 € 2,65% 1,70 1,95%

CustomW_Java_G_3


Sub-period #3 – Updated audience and new interests (new settings)

Post


Conversion

event

location

Daily

budget

Schedule

start

Schedule

end

Custom

audience


Detailed

targeting

Device

types

Optimisation for

advert delivery

Bid

amount

CustomW_Java_G_3

Website or

Messenger

-> Purchase

Daily

budget

Dec 1 Dec 15

Visits 30

days No

Conv_2

PT 21-35 M/F PT, EN1,

EN2

New set of

interests

Desktop

only




All of the previous settings used in CustomW_Java_G_2 (conversion type, daily budget,

schedule start, schedule end, languages, device types, bid amount, age and optimisation for ad

delivery) were maintained except detailed targeting and custom audience. The new custom

audience was “Visits 30 days No Conv_3” – updated group of users who visited

https://landing.jobs in the last 30 days (1 month after Visits 30 days No Conv_1 started running)

but who did not apply to any job opening).

Detailed targeting: set of interests: “IT Jobs”; “IT Jobs in Portugal”; “IT Job opportunities”;

“Oportunidades na área das TI”. Rationale: With this set of interests, the target audience will

be made of people who visited https://landing.jobs in the last 30 days before the ad started to

run, did not apply to any job opening (new set of users) and also displayed an interest Facebook

pages and ads who are in some way related with the topics “IT Jobs”; “IT Jobs in Portugal”;

“IT Job opportunities”; “Oportunidades na área das TI”. I decided to not include in the same

test a mix of that already limited audience (website visitors) with people who display interested

in pages related in some way with “IT Jobs”, “IT Jobs in Portugal”, “IT Job opportunities”,

“Oportunidades na área das TI” and also “Java”, “Java development” and “Back-end

development” because the target audience would then become too niche, thus possibly

eliminating the referral reward effect on attracting people who may know other people who

may be suited for the job.




Sub-period #3 – Updated audience and new interests (results)

Post


Results (conversions) Cost (per conversion) Budget Periodicity Reach Impressions Clicks Avg. CPM Avg. CPC CTR (%) Frequency cap CR (%)

CustomW_Java_G_3 21 0,36 € 7,50 € Daily 1210 1777 58 4,221 € 0,1293 € 3,26% 1,47 1,74%

Appendix A tech professionals (developers, designers and ... · NOVA – School Of Business And...

Documents

Transcript of Appendix A tech professionals (developers, designers and ... · NOVA – School Of Business And...