Creating App Grabber for Samsung TV

GonzaCass · May 23rd 2024

We are trying to implement a screen capture functionality on a Samsung TV with Tizen OS and send the captured image to Hyperion using WebSocket. Despite multiple attempts, we are encountering issues with the JSON data validation on the Hyperion server. Here are the steps and problems we've faced:

Capture Screen on Tizen TV:
- We are capturing a specific area of the screen using a <canvas> element.
- We convert the canvas content to a Base64 encoded PNG image.
Send Image to Hyperion:
- We establish a WebSocket connection to the Hyperion server.
- We send the image data along with the required parameters (width, height, imagedata, etc.) as a JSON object.
Errors Encountered:
- The Hyperion server returns errors indicating issues with the JSON data validation.
- Specifically, the errors mention that height and width have "no schema definition" and imagedata is expected to be a string.
- When we correct these, we get an error saying: "Size of image data does not match with the width and height".

Current JSON Payload

Here is an example of the JSON data we are sending to Hyperion:

Code

{

"command": "image",

"imagedata": "iVBORw0KGgoAAAANSUhEUgAAAHgAAAB4CAYAAAA5ZDbSAAAAAXNSR0IArs4c6QAACCxJREFUeF7tmleIVE0QhWvNYo6I...",

"width": 120,

"height": 120,

"format": "auto",

"origin": "TizenApp",

"priority": 50,

"duration": 0

}

Display More

Hyperion Server Logs

The logs on the Hyperion server show the following errors:

Code

2024-05-23T02:09:04.887Z [WEBSOCKET] (ERROR) While validating schema against json data of 'JsonRpc@::ffff:192.168.88.101':[root].height: no schema definition
2024-05-23T02:09:04.887Z [WEBSOCKET] (ERROR) While validating schema against json data of 'JsonRpc@::ffff:192.168.88.101':[root].width: no schema definition
2024-05-23T02:09:04.887Z [WEBSOCKET] (ERROR) While validating schema against json data of 'JsonRpc@::ffff:192.168.88.101':[root].format: Unknown enum value (allowed values are: ["auto"])
2024-05-23T02:09:04.887Z [WEBSOCKET] (ERROR) While validating schema against json data of 'JsonRpc@::ffff:192.168.88.101':[root].imagedata: string expected

When we fix the above errors, we encounter another error:

Code

"Size of image data does not match with the width and height"

Here is the current JavaScript code we are using to capture the screen and send the image to Hyperion:

Code

function captureScreen() {
    const captureArea = document.getElementById('captureArea');
    const canvas = document.createElement('canvas');
    const context = canvas.getContext('2d');

    const fixedWidth = 120;
    const fixedHeight = 120;

    canvas.width = fixedWidth;
    canvas.height = fixedHeight;

    const colors = ['black', 'red', 'green', 'blue', 'yellow', 'purple', 'orange'];
    const randomColor = colors[Math.floor(Math.random() * colors.length)];

    context.fillStyle = randomColor;
    context.fillRect(0, 0, canvas.width, canvas.height);

    const text = captureArea.innerText || captureArea.textContent;
    context.font = '30px Arial';
    context.fillStyle = 'white';
    context.textAlign = 'center';
    context.textBaseline = 'middle';
    context.fillText(text, canvas.width / 2, canvas.height / 2);

    const imageData = canvas.toDataURL('image/png').split(',')[1];

    sendToHyperion(imageData, canvas.width, canvas.height);
}

function sendToHyperion(imageData, width, height) {
    const ipHyperion = '192.168.88.101';
    const puertoHyperion = 8090;

    const jsonData = {
        command: "image",
        imagedata: imageData,
        width: width,
        height: height,
        format: "auto",
        origin: "TizenApp",
        priority: 50,
        duration: 0
    };

    const ws = new WebSocket(`ws://${ipHyperion}:${puertoHyperion}/jsonrpc`);

    ws.onopen = function () {
        ws.send(JSON.stringify(jsonData));
    };

    ws.onmessage = function (event) {
        console.log('Message from server: ' + event.data);
    };

    ws.onerror = function (error) {
        console.log('WebSocket error: ' + error.message);
    };

    ws.onclose = function () {
        console.log('WebSocket connection closed');
    };
}

Display More

Need assistance with the following:

Correcting the JSON schema to match what Hyperion expects.
Ensuring that the image data size matches the specified width and height.

Any guidance or examples would be greatly appreciated!

Thanks in advance!

Lord-Grey · May 23rd 2024

GonzaCass Great project which you startet.

The schema for 2.0.16 is this one: https://github.com/hyperion-pr…_schema/schema-image.json

You are missing the "image" prefix before width and height.
I would recommend that leaving the width and height out and let hyperion determine the size.
To force an output size you can provide it via scale.
I guess you wanted to have "scale": 120.

For 2.0.17-beta1 you can also provide the instances as an array the image should be applied too. You can give ["all"] for all instances.

hyperion.ng/libsrc/api/JSONRPC_schema/schema-image.json at master · hyperion-project/hyperion.ng

The successor to Hyperion aka Hyperion Next Generation - hyperion-project/hyperion.ng

github.com

In addition, if you set "duration": 0, the image will not display, as the image effect will immediately stop.

Provide duration = -1 for endless.

In general, you might want to revisit the WebSocket approach fro streaming images to Hyperion.
Other clients make use of a flatbuffer feed which will always be applied to all instances.
In this postyou find the Hyperion flatbuffer definitions.

GonzaCass · May 23rd 2024

Lol I dont know why I made a reply and it get deleted.

Well, here we go again:

Thanks for your reply Lord-Grey. I was able to send correctly the JSON and I tested with a youtube video and it works and without any latency!

Unfortunately, I made this with web app template on Tizen Studio and doing a lot of research it seems like I cant capture content outside this webapp. So, I'm doing more research now on how to capture the content of the screen even with the app closed, the real problem.

Those flatbuffers are another way from sending the content right? I'll take a look to see if I can use those.

In case that someone wants to try what I achieve, here is the code, I recommend to run it in Tizen Studio since I made it there.

main.js:

Code

// Main
document.addEventListener('DOMContentLoaded', function () {
log('Document loaded and DOM fully parsed');
document.getElementById('startCapture').addEventListener('click', startCapture);
document.getElementById('remoteButton').addEventListener('click', startCapture);
document.getElementById('testImageButton').addEventListener('click', sendTestImage);
document.getElementById('sendColorButton').addEventListener('click', sendRandomColor);
document.getElementById('startVideoCaptureButton').addEventListener('click', startVideoCapture);
document.getElementById('stopVideoCaptureButton').addEventListener('click', stopVideoCapture);


document.addEventListener('keydown', function(event) {
switch (event.key) {
case 'Enter':
log('Remote control button pressed');
document.activeElement.click();
break;
case 'ArrowUp':
focusPreviousElement();
break;
case 'ArrowDown':
focusNextElement();
break;
}
});


initializeWebSocket();
});


function focusPreviousElement() {
let focusable = document.querySelectorAll('[tabindex]');
let index = Array.prototype.indexOf.call(focusable, document.activeElement);
if (index > 0) {
focusable[index - 1].focus();
}
}


function focusNextElement() {
let focusable = document.querySelectorAll('[tabindex]');
let index = Array.prototype.indexOf.call(focusable, document.activeElement);
if (index < focusable.length - 1) {
focusable[index + 1].focus();
}
}


function startCapture() {
log('Start Capture button clicked');
captureScreen();
}


function captureScreen() {
log('Capturing screen');
const captureArea = document.getElementById('captureArea');
const canvas = document.createElement('canvas');
const context = canvas.getContext('2d');


const fixedWidth = 120;
const fixedHeight = 120;


canvas.width = fixedWidth;
canvas.height = fixedHeight;


try {
// Colores alternativos para asegurar que la imagen no sea negra
const colors = ['red', 'green', 'blue', 'yellow', 'purple', 'orange'];
const randomColor = colors[Math.floor(Math.random() * colors.length)];


context.fillStyle = randomColor;
context.fillRect(0, 0, canvas.width, canvas.height);


const text = captureArea.innerText || captureArea.textContent;
context.font = '30px Arial';
context.fillStyle = 'white';
context.textAlign = 'center';
context.textBaseline = 'middle';
context.fillText(text, canvas.width / 2, canvas.height / 2);


const imageData = canvas.toDataURL('image/jpeg', 0.03).split(',')[1]; // Lower quality to 0.03


log('Screenshot captured, Base64 length: ' + imageData.length);


sendToHyperion(imageData);
} catch (error) {
log('Error capturing screen: ' + error);
}
}


function sendTestImage() {
log('Sending test image to Hyperion');
const testImageUrl = 'https://eligeeducar.cl/content/uploads/2019/10/Imagen-que-dice-rojo-sobre-un-fondo-azul-y-azul-sobre-un-fondo-rojo--1920x550.jpg';


fetch(testImageUrl)
.then(response => response.blob())
.then(blob => compressImage(blob))
.then(imageData => sendToHyperion(imageData))
.catch(error => {
log('Error fetching test image: ' + error);
});
}


function compressImage(blob) {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onload = function () {
const img = new Image();
img.onload = function () {
const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');
const maxWidth = 120;
const maxHeight = 120;
let width = img.width;
let height = img.height;


if (width > height) {
if (width > maxWidth) {
height *= maxWidth / width;
width = maxWidth;
}
} else {
if (height > maxHeight) {
width *= maxHeight / height;
height = maxHeight;
}
}
canvas.width = width;
canvas.height = height;
ctx.drawImage(img, 0, 0, width, height);
const imageData = canvas.toDataURL('image/jpeg', 0.03).split(',')[1]; // Lower quality to 0.03
log('Compressed image captured, Base64 length: ' + imageData.length);
resolve(imageData);
};
img.onerror = reject;
img.src = reader.result;
};
reader.onerror = reject;
reader.readAsDataURL(blob);
});
}


let ws;
let messageQueue = [];


function sendToHyperion(imageData) {
const jsonData = {
command: "image",
imagedata: imageData,
name: "TizenScreenCapture",
format: "auto",
origin: "TizenApp",
priority: 50,
duration: 30000 // Duración de 30 segundos
};


log('Sending screenshot to Hyperion');


if (ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify(jsonData));
} else {
messageQueue.push(JSON.stringify(jsonData));
}
}


function initializeWebSocket() {
const ipHyperion = '192.168.88.101';
const puertoHyperion = 8090;


ws = new WebSocket(`ws://${ipHyperion}:${puertoHyperion}/jsonrpc`);


ws.onopen = function () {
log('WebSocket connection opened');
while (messageQueue.length > 0) {
ws.send(messageQueue.shift());
}
};


ws.onmessage = function (event) {
log('Message from server: ' + event.data);
};


ws.onerror = function (error) {
log('WebSocket error: ' + error.message);
};


ws.onclose = function () {
log('WebSocket connection closed');
// Intentar reconectar después de 1 segundo
setTimeout(initializeWebSocket, 1000);
};
}


function sendRandomColor() {
const ipHyperion = '192.168.88.101';
const puertoHyperion = 8090;


const colors = [
[255, 0, 0],    // Red
[0, 255, 0],    // Green
[0, 0, 255],    // Blue
[255, 255, 0],  // Yellow
[255, 0, 255],  // Magenta
[0, 255, 255]   // Cyan
];


const randomColor = colors[Math.floor(Math.random() * colors.length)];


const jsonData = {
command: "color",
color: randomColor,
duration: 120000,  // 2 minutes
priority: 20,
origin: "TizenApp"
};


log('Sending color to Hyperion: ' + JSON.stringify(jsonData));


fetch(`http://${ipHyperion}:${puertoHyperion}/json-rpc`, {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify(jsonData)
})
.then(response => response.json())
.then(data => log('Response from Hyperion: ' + JSON.stringify(data)))
.catch(error => log('Error sending color to Hyperion: ' + error));
}


// Configuración del reproductor de YouTube
let player;
function onYouTubeIframeAPIReady() {
player = new YT.Player('player', {
height: '390',
width: '640',
videoId: 'Gt6wKDnG0xA',
events: {
'onReady': onPlayerReady,
'onStateChange': onPlayerStateChange
}
});
}


let captureInterval;
function startVideoCapture() {
if (!player || !player.getIframe) {
log('YouTube Player not ready');
return;
}


log('Starting video capture');
const iframe = player.getIframe();
const videoElement = iframe.contentWindow.document.querySelector('video');
if (!videoElement) {
log('Video element not found');
return;
}


const canvas = document.createElement('canvas');
const context = canvas.getContext('2d');


canvas.width = 120;
canvas.height = 120;


captureInterval = setInterval(() => {
try {
context.drawImage(videoElement, 0, 0, canvas.width, canvas.height);
const imageData = canvas.toDataURL('image/jpeg', 0.03).split(',')[1]; // Lower quality to 0.03
sendToHyperion(imageData);
} catch (error) {
log('Error capturing video frame: ' + error);
clearInterval(captureInterval);
}
}, 100); // Capturar cada 100 ms
}


function stopVideoCapture() {
log('Stopping video capture');
clearInterval(captureInterval);
}


function onPlayerReady(event) {
event.target.playVideo();
}


function onPlayerStateChange(event) {
if (event.data == YT.PlayerState.PLAYING) {
log('YouTube Player playing');
startVideoCapture();
} else {
log('YouTube Player paused or ended');
stopVideoCapture();
}
}


function log(message) {
const logs = document.getElementById('logs');
const logMessage = document.createElement('p');
logMessage.textContent = message;
logs.appendChild(logMessage);
console.log(message);
}

Display More

TPmodding · May 24th 2024

Quote from GonzaCass

Lol I dont know why I made a reply and it get deleted.

Sometimes the forum software claims it as spam...nothing to do with you...

[APP] Hyperion (Ambilight) - SamyGO

Do you know this?

GonzaCass · May 25th 2024

Thanks for you reply TPmodding, yes I saw that but unfortunately I have a QN90C model with last firmware so I'm unable to root to get that working, at least for now.

eherranzr · Aug 21st 2024

Hi GonzaCass ,

Rebumping this post as it seems that you were still trying to get this working. Were you able to do it?

Really interested!

Creating App Grabber for Samsung TV

Current JSON Payload

Hyperion Server Logs

Participate now!

Tags