An algorithm can analyze photos and apply different artistic styles to them. Algorithms are also being used to detect emotions in text, translate between languages, and predict which social media posts will be more popular. However, algorithms also have limitations and risks, like introducing biases. Overall, algorithms can assist with creativity but human judgment is still important.

2. Balancing Human
Creativity with
Algorithms
Marshall Sponder
Lecturer, Zicklin School of Business & Associate
Professor, Rutgers Business School
Global Artificial Intelligence Conference

3. What is covered in this presentation?
1. Introduction
2. Darker Side
of Algorithms
3. Algorithms
and Creativity
4. Textbook &
Summary

4. Introduction

5. Here today to speak about a way
of humanizing Big Data and
Algorithms
Marshall Sponder develops and teaches
online and hybrid courses at Zicklin School
of Business and Rutgers University where
he holds a dual appointment.
At Zicklin, he teaches Web Analytics
courses while at Rutgers he teaches an
online class called Social Media for the
Arts. Marshall is the author of Social
Media Analytics (McGraw-Hill 2011) and
Digital Analytics for Marketing (Routledge,
2017).
Marshall is a Board Member Emeritus at
the Web Analytics Association, now called
the DAA.

6. The Digital Marketing Major is part
of the success story of Baruch

7. Digital Marketing CORE courses and some ELECTIVES at Zicklin School of Business

8. Social Media for the Arts is an Online, Asynchronous
learning course I author and teach at Rutgers University –
this semester there is 1436 students and is growing
exponentially every year.

9. Let’s start with
the Darker Side of Algorithms

10. Uber Surge Pricing generated A
$1,100 Bill For an Hour-Long Ride
http://uproxx.com/news/uber-surge-price-new-years-eve/
To this day no one
understands what is
actually going on inside
the Uber Surge
Algorithm, though
several experts have an
approximate idea of
how it works.

11. Instagram uses ‘I
will rape you’
threat as a
Facebook ad due
to a Facebook
algorithm
http://metro.co.uk/2017/09/23/instagram-uses-i-will-rape-you-threat-as-a-facebook-ad-due-to-algorithm-6949929/
The trouble is Facebook’s business model is
structurally identical whether advertisers are
selling shoes, politics or fake diet pills, and
whether they’re going after new moms, dog
lovers or neo-Nazis. The algorithms don’t
know the difference, and Facebook’s
customers are not its users.

12. Algorithms can be as biased as their trainers. Google
is more likely to advertise executive-level salaried
positions in the UK to search engine users if it thinks
the user is male, according to a Carnegie Mellon study.
Photograph: Yui Mok/PA
https://www.theguardian.co
m/technology/2016/aug/03
/algorithm-racist-human-
employers-work#img-2

13. An algorithm that
automatically delivers a
‘don’t interview’ verdict
to candidates with
criminal records
disproportionately
impacts black job
seekers.
https://www.theguardian.com/technology/2016/aug/03/algori
thm-racist-human-employers-work#img-2

14. One of the challenges in algorithm-based hiring is that
currently there is no standard way to measure the
outcome of an algorithm’s choices.
https://www.theguardian.com/technology/2016/aug/03/algori
thm-racist-human-employers-work#img-2

15. But it is Important to
get algorithms right
as they are running
everything now -
Many prison
inmates' futures
depend on racially
biased algorithms
http://theweek.com/articles/627570/many-prison-inmates-futures-depend-racially-biased-algorithms

16. There is Hope: scientists have devised a way to
test whether an algorithm is introducing
gender or racial biases into decision-making.
https://www.theguardian.c
om/technology/2016/dec/
19/discrimination-by-
algorithm-scientists-devise-
test-to-detect-ai-bias

17. Police use a Geo-tracking Algorithmic Surveillance Tool,
Geofeedia, to Scan Social Media for criminal activity.
https://www.nytimes.com/2016/10/12/technology/aclu-facebook-twitter-instagram-geofeedia.html

18. Algorithms are best viewed in terms of expected inputs vs. expected
outputs, especially when the programming code can not be examined.
Outputs that are several standard deviation above or below the mean or average (such as a
1000.00 cab ride that normally costs 100.00) would be a red flag that there is something wrong
or abnormal about what the program (algorithm) is doing. Even without programming
knowledge, anyone can begin to examine algorithms in this way.
https://en.wikipedia.org/wiki/Black_box

19. Watch how you handle big data, FTC warns businesses -
'We need systems for auditing the proprietary
algorithms,' an ACLU attorney says
https://www.computerworld.com/article/3019911/big-data/watch-how-you-handle-big-data-ftc-warns-businesses.html

20. How to balance Algorithms
with human creativity and
keep the best of both!

21. Understanding Algorithms
• Algorithms are a series of instructions that
are used to find patterns within a set of
data and make decisions upon that data.
• Computer and humans run algorithms.
• Many of the algorithms run on computers
are a “black box” and can not be examined
closely.
https://piktochart.com/wp-content/uploads/2014/12/Steps-to-choose-an-Infographic.jpeg

22. In Music – Algorithms are beginning to choose the
next hit songs, and perhaps, predetermine them.
A Machine Successfully Predicted the Hit Dance Songs of 2015
https://motherboard.vice.com/en_us/article/bmvxvm/a-machine-successfully-predicted-the-hit-dance-songs-of-2015
Each song is encoded into
various measurements that
can be operated on by
specific algorithms and
plotted, as show in the chart
on this slide.
The algorithm makes a
prediction of the likelihood
a song will be a Hit based on
it’s position on the cartesian
graph, and its calculated
score.

23. A Machine
Successfully
Predicted
probabilities
connected with Hit
Dance Songs of 2015,
and is probably
selecting even more
hits in 2018
https://motherboard.vice.com/en_us/article/bmvxvm/a-
machine-successfully-predicted-the-hit-dance-songs-of-2015

24. The
Washington
Post’s robot
reporter has
published 850
articles in the
past year –
like this one
https://digiday.com/media/washington-posts-robot-reporter-published-500-articles-last-year/?utm_medium=email&utm_campaign=digidaydis&utm_source=publishing&utm_content=170914

25. Creating Computer
Vision and Machine
Learning Algorithms
That Can Analyze
Works of Art at
Rutgers University.
https://www.mathworks.com/company/newsletters/articles/creating-computer-vision-and-machine-learning-algorithms-that-can-analyze-works-of-art.html?refresh=true&requestedDomain=www.mathworks.com

26. The creativity algorithms were tested
on two data sets containing more
than 62,000 paintings. The algorithm
gave high scores to several works
recognized by art historians as both
novel and influential, including some
of the works shown in Figure 3.
Ranking even higher than Pablo
Picasso’s “Young Ladies of Avignon”
(1907) in the same period were
several paintings by Kazimir Malevich.
This result initially surprised me, as I
knew little about Malevich’s work.
The implications suggest that the art
market, including historical art, may
be over-rated for some artists, and
under valued, for others.
https://www.mathworks.com/company/newsletters/articles/creating-computer-vision-and-machine-learning-algorithms-that-can-analyze-works-of-art.html?refresh=true&requestedDomain=www.mathworks.com

27. Exploring world-
wide clothing
styles from
millions of photos
–@ Cornell
University
https://arxiv.org/pdf/1706.01869.pdf

28. https://arxiv.org/pdf/1706.01869.pdf

29. Amazing New
Algorithms Will
Fix Your Photos
Before You
Even Take
Them
http://www.sciencealert.com/new-algorithms-can-fix-your-photos-before-you-even-take-them

30. Google’s Algorithmic Photographers
Are Almost as Good as the Real Deal
https://flipboard.com/@flipboard/-
googles-algorithmic-photographers-
are-a/f-e848f1e2d3%2Fvice.com

31. New
algorithm
gives photos
Picasso-style
makeovers
http://mashable.com/2015/08/29/computer-photos/#CBRQjLJfrOqc

32. This Apps'
Creations Sure
Look Like
Masterworks, But
Is It Art?
https://www.instagram.com/p/BIulqIpD2oL/
http://www.npr.org/sections/alltechconsidered/2016/09/04/492408169/these-apps-creations-sure-look-like-masterworks-but-is-it-art

33. Algorithms
can turn
sketches into
art with
machine
learning
(Vincent AI)
https://www.cambridgeconsultants.com/vincent

34. Search Engines such as Bing have Automated Image
Detection In Visual Search Queries – searchers can
also use semantic search keywords (supplied) to dig
deeper and find images with specific elements in it.
https://www.mediapost.com/publications/article/307695/

35. Impressive Adobe Algorithm Transfers One Photo’s Style Onto Another
https://petapixel.com/2017/03/29/cornelladobe-show-copy-color-lighting-one-photo-another/

36. Google releases
smart image
analysis tools to
let robots
recognize
images, text
https://www.pcworld.com/article/3034967/data-center-cloud/google-releases-intelligent-image-analysis-tools-for-developers.html

37. Google has a Cloud Vision API (several others have similar offerings) that is easy to build applications around.
I put the building
I just moved into,
based on a photo
I took recently,
and the Google
Cloud Vision API
did a pretty good
job of decoding
it’s meaning.
https://cloud.google.com/vision/

38. Text Analytics platforms have been successfully employed, piggybacking
off algorithms to detect emotions in text (and images), then isolate it –
image taken from a Crimson Hexagon analysis.

39. Algorithms can
detect the most
interesting parts of
an image and how
memorable it is.
http://memorability.csail.mit.edu/demo.html

40. Geodata can be
easily surfaced,
algorithmically
clustered and
charted for
instant insights
and futher
analysis.

41. Microsoft
researchers
are teaching
AI to write
stories about
groups of
photos
https://venturebeat.com/2016/04/14/microsoft-ai-visual-storytelling/

42. Google created Zero-Shot Translation with a Multilingual Neural Machine Translation System –
Computers are teaching themselves (deep learning) how to translate between languages.
https://www.tnooz.com/article/google-translation-update/

43. An Algorithm can project a
3D face from a 2D image –
any image.
http://www.cs.nott.ac.uk/~psxasj/3dme/view.php?name=59bf2f00d5615
http://www.cs.nott.ac.uk/~psxasj/3dme/index.php

44. The iPhone X
will open
new realms
for creatives
FaceID still scares many and there is a lot of disagreement about the wisdom of putting it in a
phone, still, but ultimately is the natural evolution of where we have been going all along.

45. Creativity is not limited to the visual arts, this Robot Therapist Talks to Patients Via Facebook
- Woebot is available 24/7 via Facebook.
https://www.woebot.io /

46. With Algorithms
doing so much for
us, is there any
point in being
creative? I used to
paint, but should I
still bother?
https://www.linkedin.com/pulse/warmth-caring-relationships-age-robots-algorithms-ai-bill-murphy

47. 30 years ago, I painted my “Homage to
Manet” painting, and struggled, but it’s
hard for me to imagine doing the same
thing today. I’d rather write and teach.
Maybe I’ll go back to painting again
someday, when I figure out why I need
to.

48. Last year I used the Prisma app to reimage the
Manhattan skyline, this is not a filter, but an
actual algorithm repainting a scene – I used to
spend hours, days, trying to paint the same.

49. For LinkedIn, use third-party algorithms
such as snappr.co/photo-analyzer to
choose your best photos to use for the
profile image. LinkedIn profile pictures
are important aspect of a member’s
profile, and can be a deciding factor for
jobs or other professional
opportunities. In this case, we’re using
algorithms to help put our best face
forward. The same program can be
used for any photo.

50. Everypixel is neural network if your photo is good or not and can be employed with any image media.
Note: using different algorithms on the same photo will produce different results.
https://everypixel.com/aesthetics

51. Use a Predictive Algorithm to determine the best Tweets to post.
The Retweeted More Algorithm successfully predicted which of two
@realdonaldtrump tweets on the same subject would be more viral.
https://chenhaot.com/retweetedmore/

52. Finding the
right things
to do and
see when
traveling
using
Google
Trips
algorithm
https://get.google.com/trips/

53. EDM artists can use
algorithms such as the
Dance Hit Predictor tool to
pick their best
compositions to promote.
http://dorienherremans.com/dance/uploader.php

54. Algorithms
can apply an
artists
personal
style to 3D
objects in
real time
http://stylit.org/

55. Use algorithms such as LaMem
to curate your best Instagram,
Facebook, Pinterest and
SnapChat photos to share in
Social Media, particularly for
paid campaigns.
https://www.instagram.com/webmetricsguru/

56. Use algorithms such as LaMem
to curate your best Instagram,
Facebook, Pinterest and
SnapChat photos to share in
Social Media, particularly for
paid campaigns.
https://www.instagram.com/webmetricsguru/
score: 0.689 score: 0.281 score: 0.545
score: 0.58 score: 0.81 score: 0.618
score: 0.551 score: 0.645 score: 0.745
score: 0.738 score: 0.545 score: 0.606

57. Use the Google Cloud API to better annotate the image with the
best descriptors and keywords – these are likely to be favored by
Google and Social Media Text Analytics engines.
Finish up by coming up with a good tagline for the post with an
algorithm such as https://chenhaot.com/retweetedmore/, explored
in an earlier slide. Finally, post the image and it’s annotation in
Social Media.

58. Use DeepArt to take make favorite artwork (painting) and turn it into a
style that an algorithm uses to repaint any other image it is applied to.
One of my
best
paintings
from 1988
The
building I
live in.

59. An Algorithm took my original photo and repainted it
in the style of my 1988 painting – giving me additional creative ideas.
+ =

60. What if I wanted to know what Homage to
Manet, would have looked like if Manet had
painted it, instead of me? ( I suppose, a
“Homage” to himself)? – ha!
I used Manet’s “A Bar at the Folies-Bergere”
that is pictured in my painting as the filter!
Translating my Homage to Manet into a Manet Style
painting did not work out that well for my painting –
maybe the “brush” being applied, was wrong.
I guess, not all things we think about, would really
work, but at least, now we know.

61. Assuming Vincent AI becomes a Tablet App (for the iPad, for example) then various styles
can be applied to render it.
Ideally, it would be better for artists to program or load in their own styles, and then
combine them with what the algorithm (Vincent AI) renders – interacting in a back and forth
dance with the algorithm, till the artist gets the rendering they want.

62. Use Bing Visual
Search to find
extremely
specific
imagery to use
for inspiration
in painting,
photography
and fashion.

63. Artists and Illustrators often fail to
visualize and convey in drawing and
painting the complex geometry and
volume of the face when they use
photographs.
Using the algorithm from the 3D
Face Reconstruction augment to
recreate a 3D model of the face,
the depth and foreshortening
inherit in photographs can
compensated for, creating a better
model for the portrait.
The angle can be changed, as
needed, by the artist, up to the
level of what the program allows.
https://en.wikipedia.org/wiki/Abraham_Lincoln#/media/File:Abraham_Lincoln_O-77_matte_collodion_print.jpg

64. Use the various Vision APIs to generate Metadata
around your visual content. Search engines already
examine images on webpages and examine if the
meanings derived from the images match up with
the textual descriptions on the page.
Similarly, creators can put in images that captivate
them and use the Vision APIs to come up with
descriptors, and tune them for SEO, thereby raising
their visibility online and creating a better user
experience.
This image was pulled in real time from
Picodash.com, a geolocation filtering platform for
Instagram, looking at the Metropolitan Museum of
Art

65. Use Algorithms to your advantage – do something creative with the metadata.

66. Use the metadata to create
better creative! This fuels
SEO & Social Media.
My Story using the Algorithm:
A young woman wearing glasses walked into
the European Painting gallery at the Met and
was overjoyed as she stood in front of a
Venus and Adonis painting by either Pieve di
Cadore or Titian painted in Venice during the
Renaissance.
Like the Mona Lisa, there is a mysterious
message being communicated to the
onlooker. We can only guess and what this
mysterious visitor to the MET is trying to say.
Have fun with Algorithms, use them, instead of letting them use you!

67. What is the green tea
mcflurry?
• Back in 2013, McDonald’s
Japan released the Matcha
Oreo McFlurry, which blended
rich vanilla ice cream with
powdered green tea and
crunchy chocolate biscuit
pieces.
• The mix was so popular it
returned for an encore season
in 2014, and while it was
missing from menus last year,
the Matcha McFlurry has
finally returned with a whole
new look, featuring more
traditional Japanese flavors.
Use geolocation
to explore …
anything, find
the audience
and target.
http://en.rocketnews24.com/2016/03/23/mcdonalds-japan-releases-brand-new-matcha-mcflurry-for-a-limited-time

68. Found the liked Instagram post and where it
was taken then plugged the
latitude/longitude code into Google can
resolve the exact location and audience
nearby.

69. Use the data to
iterate, explore, find
the story (used
Picodash to do this).
Then, use the other
approaches outlined
here to piggyback off
of the algorithms
and combine them
creatively.

70. Using Geolocation or a precise location together with emotion tracking can give us a better idea of
what people are feeling at the price moment they are touching our content and marketing.
Granted, these technologies are still niche and leading edge (and SCARY), but sooner or later they
will become more accessible and easier to work with. Yes, there are scary elements to all of this,
but we don’t have to reject these tools outright, we can humanize them, and run them, instead of
letting them run us.
http://mashable.com/2017/07/27/disney-facial-recognition-prediction-movies/#hm1qXklrxmqq

71. I ran a Facebook
Ad to increase the
enrollment of the
Fall 2017 Social
Media for the Arts
course – I know my
audience well.

72. I ran a
Facebook Ad to
increase the
enrollment of
the Fall 2017
Social Media
for the Arts
course (by as
much as 50
students)
With the right approach I made these platforms and algorithms work for me!

73. Summary -1
1. Most Algorithms are trained
by humans and learn from
them, inheriting their bias.
2. Algorithms are capable of
creativity and often match
aspects of our own creativity.
3. Creatives can use Algorithms
to increase the effectiveness
and scope of their Art.

74. Summary -2
4. The right workflow solutions for artists
have not emerged yet.
5. The tools that exist are too geared to
marketers who have short term goals and
are probably too expensive to be useful.
6. Most of the existing software is not
extensible or user friendly - most remain
at the proof of concept level.

75. New Textbook
• www.routledge.com/9781138190672
• http://bit.ly/DAFM_MS (published 10/4/17)
• amazon.com/author/marshallsponder
• http://www.linkedin.com/in/marshallsponder
• @webmetricsguru

76. Thank You!
Marshall Sponder
Lecturer, Zicklin School of Business
Associate Professor, Rutgers Business School
New Book - http://bit.ly/DAFM_MS