What is AI Data Sculpture?
last updated: 2021-06-13 first posted: some time in 2019, I don’t remember the exact date.
Simplified view of Reality
Why the term “AI sculpture” or “AI Data Sculpture” feels weird; let’s dive into that now.
-> Side kick: In my previous post titled How do you make music with Artificial Intelligence?, I give a brief introduction and suggest readings about AI. I am passing those discussions here. The type of AI these artists and I are using in their work is Applied Artificial Intelligence (autonomous software), not General Artificial Intelligence (the kind that would conquer the world).
Some prefer the term autonomous agents instead of AI. The term AI is problematic because it creates this notion that there is an intelligence related to the autonomous software, without clarifying what intelligence is. Although we may percieve some behaviours that are generated by the software as similar to behaviour of us—intelligent beings, it is arguable to call these software “intelligent.”
...(*￣０￣)ノ
Back to the term AI Data Sculpture...
(＃`Д´) AN OBJECT IS NOT AN AGENT....!!!!
Ohommm... I am calm now.
The simplified view of reality has three components: agents, objects, and the environment. Agents and objects situate in the environment. Objects are static, they do not react to the changes in the environment, they do not interact with other objects. Agents can act on the enviroment, react to the changes in the environment, and interact with other agents.
Let me explain reactivity and interactivity briefly. First, let’s give an example of reactive behaviour using house cleaning. Let’s say that you are a clean and tidy person. You (agent) desire to keep your house clean. Your house (environment) has become dirty (changes in the environment), you (agent) realize this at some point (agent’s perception). Since you (agent) intend to live in a clean house, you (agent) plan to clean your house. Your cleaning plan includes actions, such as preparing the cleaning supplies, cleaning one room at a time etc.
What would be interactivity then? Interactive behaviour happens between agents. Let’s say that you start cleaning your house. You move a piece of furniture, and there is a mouse running out. You immediately stop cleaning and get out of the house and keep running...☆ﾐ(o*･ω･)ﾉ
By jumping at you, the mouse has changed your actions that are planned previously. By moving the furniture, you made a change in the behaviour of the mouse, and it run away from you to find another hiding spot. Notice that this is different than reactivity. After you see the mouse, you don’t give a heck about the cleaning anymore, you just run away full speed. Both agents actions are affected by each other.
When we mention the term sculpture, my immediate connotation is an object. You may think that the examples of kinetic sculptures contradict the idea that a sculpture is an object. Although kinetic sculptures are moving and dynamic, they do not exhibit agent behaviours such as proactivity, reactivity, or interactivity. When the artwork exhibits reactive and interactive behaviours, then we move towards the realm of Interactive Arts, and the discussion of agent behaviours. In the case of many artworks that are labeled as AI Sculptures, the artworks are fixed-media videos that are played on a screen, and sometimes these screens are installed on a custom-shaped object; hence appearing more like an object than an agent.
A follow-up question is, how do we label these works? Let’s brainstorm. Given that these artworks use content that is generated by an Applied AI algorithm, maybe “AI generated Data” is a better fit? Still though, public may percieve “AI generated Data” as a content that is generated by an Artificial General Intelligence with intrinsic motivations, which is not the case. Joanna Bryson---an AI ethics researcher mentions, communicating a work to the public in this manner is problematic. Another issue is, if the artwork is a “Data Sculpture”, wouldn’t we expect some relationship between the properties of the data, and the form of the sculpture? Do “AI Data Sculpture” artworks have this relationship?
PS: I started writing this article sometime in 2019, and it was online, but not listed on my website until now. Somehow, the page moved its way up on the google searches of “AI Data Sculpture.” Internet has its own marvelous universe.
The terminology of agents and the concept of simplified view of reality is based on the book:
- Wooldridge, M. (2009). An Introduction to MultiAgent Systems. John Wiley & Sons.
An extended discourse on the difference of reactivity and interactivity can be found here with a focus on interactive music systems:
- Tatar K. & Pasquier P. (2019). Musical agents: A typology and state of the art towards Musical Metacreation. Journal of New Music Research, 48(1), 56–105.
- Joanna Bryson on AI ethics: http://www.cs.bath.ac.uk/~jjb/web/ai.html
An Introduction to Artificial Intelligence for Music
Updated on 21-10-2022
I get this question a lot:
How do you make music with Artificial Intelligence?
This question goes as deep as the question “how do you make music?”; but still, I will try to give a general introduction to making music with Artificial Intelligence. There will be bunch of readings in the following, yet this is not a page that gives an explicit implementation tutorial.
Disclamier: This is an informal page. I may speak out my personal views, and speculate here and there. Yet, the publications are solid, formal, and peer reviewed. Well, maybe too formal for those who are not familiar with the academic language.
Open Access: Some papers are not open access, that is, you need a subscription to the journal to download the article. Most university libraries are subscribed to these journals and you can access papers through your library. Please contact your library if you can’t access any paper. If you don’t have any ties to an academic institute, you could just ask the authors to share an author copy with you. The contents of author copies are the same as the journal copies. The author copies are just not formatted in the journal style. You could easily send the authors an email, or send them a message on researchgate to ask for an author copy. Sometimes, the authors also share their papers on their personal websites. Open access is a known issue in academia, and many researchers are aware of the issue and proposing solutions. However, things are moving with an institutional speed, aka slow as hell.
Let’s do it.
Will AI conquer the world?
My computer can’t even find a file if I mistake one letter. Good luck on conquering the world. Pop-science articles on AI are detrimental to the society, and let’s stop sharing these clickbaits. There are other issues related to AI technologies that need immediate actions. I will touch on these under the section AI & Ethics at the bottom. Spoiler: I don’t think autonomous systems will conquer the world, simply because they lack instrinsic motivations. My computer will not wake up one day and start cooking some delicious food, unless I program it to do so. Yet, this doesn’t mean that humans will not make malicious autonomous software to cause real harm. Actually, this has happened already:
Documentary: Zero Days
Let’s clarify what we mean when we call a system AI system. Two main disciplines around artificial intelligence appear in the research:
Artificial General Intelligence (AGI) and Applied Artificial Intelligence (AAI)
Artificial General Intelligence (AGI) is quite philosophical and mainly focuses on human cognition, how we communicate etc. AGI systems try to solve multiple problems at once with one architecture, and they are mainly implemented as computational systems, aka software. That is asking the question if we could have a system that cooks, communicates, talks, plays music, choreographs dance etc.
Side kick: If you have spare time, here is a paper that compares various cognitive architectures in the AGI domain. This paper is not necessarily related to music. However, the CLARION architecture mentioned in this paper has been applied to music.
-> Thórisson, K., & Helgasson, H. (2012). Cognitive Architectures and Autonomy: A Comparative Review. Journal of Artificial General Intelligence, 3(2), 1–30. https://doi.org/10.2478/v10229-011-0015-3
Applied Artificial Intelligence (AAI) is autonomous systems solving one problem at a time. Almost 99% of AI systems you see online are in this category. These systems automatize one task in a smart way. AAI will be our main focus, and from now on, I refer to AAI when I mention artificial intelligence.
What is Musical Metacreation?
Making music with Artificial Intelligence =Creative Artificial Intelligence for Music = Musical Metacreation <- Well, sort of...
The following paper gives an introduction to the general idea of making music with AI, and Computational Creativity. -> Pasquier, P., Eigenfeldt, A., Bown, O., & Dubnov, S. (2017). An Introduction to Musical Metacreation. Computers in Entertainment, 14(2), 1–14. https://doi.org/10.1145/2930672
Looking from a Wider Perspective
Making music with AI is strongly connected with a broader discipline called Computational Creativity (CC).
CC asks the questions,
-> What is creativity?
-> What would be the definition and taxonomy of creativity?
I strongly recommend the writings of Margaret Boden if you are interested in these two questions. Here are some:
A must read -> Boden, M. A. (2015). Creativity and ALife. Artificial Life, 21(3), 354–365. https://doi.org/10.1162/ARTL_a_00176
Additionals ->Boden, M. A. (2010). The Turing test and artistic creativity. Kybernetes, 39(3), 409–413. https://doi.org/10.1108/03684921011036132
->Boden, M. A. (2009). Computer models of creativity. AI Magazine, 30(3), 23.
->Boden, M. A., & Edmonds, E. A. (2009). What is generative art? Digital Creativity, 20(1–2), 21–46. https://doi.org/10.1080/14626260902867915
-> How do we model human creativity?
-> How do we support creativity with autonomous systems?
-> How do we make artificially creative systems?
These are more broad questions. Pasquier et al. (2017) gives a nice introduction to these questions in the domain of music. The first paper in the next section also covers these questions applied to music.
Can I play music with Artificial Intelligence?
Yes. ︎We recently published a survey on interactive music systems based on artificial intelligence, aka musical agents. The topic goes deep, and the paper is quite heavy. Still, I strongly recommend the sections 1, 2, 3 for an introduction to the topic; and section 8 to understand where we are at in this area. If you have time, feel free delve into the whole paper.
-> Tatar, K., & Pasquier, P. (2018). Musical agents: A typology and state of the art towards Musical Metacreation. Journal of New Music Research, 47(4), 1–50. https://doi.org/10.1080/09298215.2018.1511736
You can download a free eprint copy of this paper using this link.
Can I compose music with Artificial Intelligence?
Yes. I prefer categorizing these systems as purely generative as they run offline and do not interact with other software agents and humans (excluding the user interaction with hyper-parameters). Following paper surveys these systems from a conventional music perspective:
-> Herremans, D., Chuan, C.-H., & Chew, E. (2017). A Functional Taxonomy of Music Generation Systems. ACM Computing Surveys, 50(5), 1–30. https://doi.org/10.1145/3108242
And, the following paper covers purely generative systems that apply Deep Learning:
-> Briot, J.-P., Hadjeres, G., & Pachet, F. (2017). Deep Learning Techniques for Music Generation-A Survey. ArXiv Preprint ArXiv:1709.01620.
I already mentioned so many papers! I recommend the following, find a specific musical task that you want to focus, and read related sections in those papers. From there, you can follow the citations in those sections to go deeper in your focus. Some systems share their codes as a link in the paper, you can download and check them out.
AI & Ethics
Following two papers cover Ethics in Artificial Intelligence for musical applications. They are still in my to-read list, and I am sure they will be a fun reading to cover.
-> A. Holzapfel, B. L. Sturm, and M. Coeckelbergh, “Ethical dimensions of music information retrieval technology,” Trans. Int. Soc. Music Information Retrieval, vol. 1, no. 1, pp. 44–55, 2018.
-> B. L. Sturm, M. Iglesias, O. Ben-Tal, M. Miron, and E. G ́omez, “Artificial intelligence and music: Open questions of copyright law and engineering praxis,” MDPI Arts, vol. 8, no. 3, 2019.
Before we finish, I mentioned earlier that I will touch on AI & Ethics in general. I think the most immediate action is required to ban AI weapons. Think about this way, companies could make a turret to automatically detect and shoot any enemy entering an area. This may sound futuristic, but here we are: wikipedia.org/wiki/SGR-A1
I think that no machine should make the decision to end a human life. There is almost a consensus in the academia about this:
-> futureoflife.org/lethal-autonomous-weapons-pledge/?cn-reloaded=1&cn-reloaded=1
-> www.theguardian.com/technology/2017/aug/20/elon-musk-killer-robots-experts-outright-ban-lethal-autonomous-weapons-war
We should take this matter as serious as nuclear weapons. Next time you are about to share another AI conquering world article, please share an article about why we should ban autonomous weapons instead.
There are also social effects of autonomous systems. I really like Johanna Bryson’s writings on that topic: http://www.cs.bath.ac.uk/~jjb/web/ai.htm.
Like all the topics I mentioned so far, AI & Ethics goes quite deep. IEEE started the discussion of standards for autonomous systems, and you can find a book on this topic here:
https://standards.ieee.org/industry-connections/ec/autonomous-systems.html
Last Words...
I feel like I talked about readings of hundreds of pages, without giving explict guidelines for the implementation of autonomous systems making music. Well, that is another post I guess.
Practical side of things
maybe start from these
AI Music Creativity conference: https://aimusiccreativity.org/
Handbook of AI for music: https://link.springer.com/book/10.1007/978-3-030-72116-9
Fundamentals of Music Processing: https://link.springer.com/book/10.1007/978-3-319-21945-5
Deep Learning book: https://www.deeplearningbook.org/
here is a full list:
Conferences:
Audio For Games
Audio Engineering Society – International Conference on Audio for Games
ACM Multimedia (ACMMM)
A venue for showcasing innovative industrial products and scientific achievements in multimedia.
ACM SIGGRAPH
Conference promoting the generation and dissemination of computer graphics and interaction techniques.
ACM Computational Aesthetics
Symposium focused on the analysis and augmentation of creative behaviours.
ArtTech
International conference for those working in the production and dissemination of digital and electronic art.
Autonomous Agents and Multiagent Systems (AAMAS)
Yearly conference for research on autonomous agents and multiagent systems.
Cognitive Science Society (CogSci)
Annual meeting of researchers working on basic and applied cognitive science research.
Creativity & Cognition (C&C)
Conference focusing on understanding, designing tools for and augmenting human creativity.
EvoStar (Evo*)
Conference centered around theoretical and applied research in genetic and evolutionary programming.
Generative Art (GA)
A meeting place for exchanging ideas, approaches and experiences in generative systems.
International Computer Music Conference (ICMC)
A themed yearly international conference in computer music research and composition.
International Conference on Computational Creativity (ICCCX)
Facilitating the exchange of ideas on computational creativity in a cross-disciplinary setting.
International Society for Music Information Retreival (ISMIR)
Conference for research in music modelling, creation, search and processing.
International Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging
A conference meant to bring together those developing computer based tools to solve aesthetic problems and those who use these tools.
International Workshop on Machine Learning and Music (MML)
Workshop held in conjunction with the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD).
New Interfaces for Musical Expression (NIME)
Forum focusing on the design of new musical interfaces.
Sound & Music Computing (SMC)
A forum for interdisciplinary exchanges on the topics of sound and music computing.
Live Interfaces (ICLI)
“The conference seeks to investigate cross-disciplinary understandings of performance technology with a particular focus on issues related to the notion of ÔlivenessÕ in interaction.”
Summer Workshop on MIR
This workshop will teach the underlying ideas, approaches, technologies, and practical design of intelligent audio systems using Music Information Retrieval (MIR) algorithms.
JournalsMusic & Sound
Computer Music Journal (CMJ)
Leading journal focusing on computer music topics and discussions.
Journal of New Music Research
Journal of New Music Research is a peer-reviewed academic journal covering research on musicology (including music theory), philosophy, psychology, acoustics, computer science, engineering, and other disciplines.
Organised Sound
International peer-reviewed journal focusing on current uses and issues in the combination of music and technology.
IEEE Trans. on Audio, Speech, and Language Processing (TASLP)
The IEEE Transactions on Audio, Speech and Language Processing covers the sciences, technologies and applications relating to the analysis, coding, enhancement, recognition and synthesis of audio, music, speech and language
Journal on Audio, Speech and Music Processing (ASMP)
The aim of EURASIP Journal on Audio, Speech, and Music Processing is to bring together researchers, scientists and engineers working on the theory and applications of the processing of various audio signals, with a specific focus on speech and music.
Journal of the Audio Engineering Society (JAES)
The Journal of the Audio Engineering Society — the official publication of the AES — is the only peer-reviewed journal devoted exclusively to audio technology
Journal of Interdisciplinary Music Studies
The Journal of Interdisciplinary Music Studies (JIMS) is an international peer-reviewed journal. Published twice per year, it aims to establish a broad interdisciplinary platform for music researchers. JIMS especially promotes collaborations between sciences and humanities and between theory and practice, and provocative submissions that stimulate interdisciplinary discussion. JIMS is unique among international music research journals in its coverage of all epistemological approaches to all musical issues.
Art & Technology
International Journal of Arts and Technology (IJART)
Journal focusing on state-of-the-art research in works bridging art and technology.
Leonardo Journal
An interdisciplinary journal working across the domains of art, science and technology.
International Journal of Performance Arts & Digital Media
Journal which focuses on the combination of digital technologies with the performance arts (theatre, dance, music, live art).
PAJ: A Journal of Performance and Art
A three-issue per year journal which charts new directions in performance, video, drama, dance, installations, media, film, and music, bringing together ideas of theatre and the visual arts.
Intelligent Systems
Autonomous Agents and Multi-Agent Systems (JAAMAS)
Journal for research into autonomous agents and multi-agent systems.
ACM Transactions on Interactive Intelligent Systems (TiiS)
This journal publishes papers on research concerning the design, realization, or evaluation of interactive systems that incorporate some form of machine intelligence. TIIS articles come from a wide range of research areas and communities. An article can take any of several complimentary views of interactive intelligent systems
International Journal of Art, Culture and Design Technologies
The mission of the International Journal of Art, Culture and Design Technologies (IJACDT) is to promote interdisciplinary research concerning the influence of technological innovation on creativity and other human issues. IJACDT bridges the humanities, artistic, scientific, and technical disciplines. It is a nexus for information exchange among academia and industry addressing theory, criticism, and practice.
Creativity
Creativity Research Journal
Presents a range of research focusing on different approaches to studying creativity.
Journal of Computing, Design and Creativity
Focused on the development of new computational systems and interactive experiences for enhancing design creativity and collaboration.
Multimedia
IEEE Trans. on Multimedia (TMM)
The IEEE Transactions on Multimedia covers the breadth of research in multimedia technology and applications, including: circuits, algorithms and architectures, software design, synchronization, joint processing of multimedia/multimodal signals/data, compression, storage, retrieval, networking, multi-modality devices/systems.
IEEE Multimedia
The magazine contains technical information covering a broad range of issues in multimedia systems and applications. Articles discuss research as well as advanced practice in hardware/software and are expected to span the range from theory to working systems. Especially encouraged are papers discussing experiences with new or advanced systems and subsystems
ACM Transactions on Multimedia Computing, Commmunications, and Applications (TOMM)
The ACM Transactions on Multimedia Computing, Communications, and Applications is the flagship publication of the ACM Special Interest Group in Multimedia (SIGMM). It focuses on multimedia computing (I/O devices, OS, storage systems, streaming media middleware, continuous media representations, media coding, media processing, etc.), multimedia communications (real-time protocols, end-to-end streaming media, resource allocation, multicast protocols, etc.), and multimedia applications (databases, distributed collaboration, video conferencing, 3D virtual environments, etc.).
Multimedia Tools and Applications
Multimedia Tools and Applications publishes original research articles on multimedia development and system support tools, and case studies of multimedia applications. Experimental and survey articles are appropriate for the journal. The journal is intended for academics, practitioners, scientists and engineers who are involved in multimedia system research, design and applications. All papers are peer reviewed.
Multimedia Systems
This journal details innovative research ideas, emerging technologies, state-of-the-art methods and tools in all aspects of multimedia computing, communication, storage, and applications. It features theoretical, experimental, and survey articles.