{"id":1016,"date":"2020-07-09T15:30:27","date_gmt":"2020-07-09T15:30:27","guid":{"rendered":"https:\/\/www.confianzit.com\/cit-blog\/?p=1016"},"modified":"2022-10-31T17:53:28","modified_gmt":"2022-10-31T17:53:28","slug":"rectangle-detection-and-perspective-correction-in-react-native","status":"publish","type":"post","link":"https:\/\/www.confianzit.com\/cit-blog\/rectangle-detection-and-perspective-correction-in-react-native\/","title":{"rendered":"Rectangle Detection and Perspective Correction in React Native"},"content":{"rendered":"\n\n[et_pb_section admin_label=”section”]\r\n\t\t\t[et_pb_row admin_label=”row”]\r\n\t\t\t\t[et_pb_column type=”4_4″][et_pb_text admin_label=”Text”]\n
Since the age of artificial intelligence and machine learning, one of the most important applications to which they have been applied is for text detection and image processing, creating the field of computer vision. While computer vision was almost arcane in the old days and developing applications related to the same required much effort and deeper understanding of many mathematical concepts, nowadays it is used almost everywhere, thanks to the development of the famous OpenCV library by Intel.<\/p>\n\n\n\n
Most of the applications for image processing can be handled well with the OpenCV library, which is commonly used with Python to handle complex cases, and the source for the library is also available in other platforms. In the context of mobile application development, OpenCV has provided libraries for both Android and iOS.<\/p>\n\n\n\n
One of the widely used applications of OpenCV is to detect a document and possibly apply perspective correction to get a clear view of the document. This is quite easy in OpenCV and we could use built-in functions such as getPerspectiveTransform(src, dst). While this works out great when developing native applications, if the app is developed in a cross-platform ecosystem using frameworks such as React Native, this won\u2019t be ideal. This is because we will have to integrate OpenCV in the native source files of the React Native project and write native code to implement the same which is used in React Native. This defeats the entire purpose of cross platform development and requires specialized knowledge in native development which brings us back to square one. Thankfully due to the massive adoption of React Native over the recent years and the open source community we now have alternatives available as plugins to use like react-native-rectangle-scanner. So, let\u2019s see how to use this plugin to implement a simple app in React Native.<\/p>\n\n\n\n
Getting Started<\/strong><\/p>\n\n\n\n
<\/figure><\/div>\n\n\n\n
Let\u2019s create a sample application to detect a rectangle document and show it to the user with the perspective correction applied as shown above.<\/p>\n\n\n\n
Using the react-native-cli tool let\u2019s initialize an empty project. This can also be done using expo and really is up to personal preference. The npx command allows you to use the latest version of the React Native.<\/p>\n\n\n\n
In the package.json<\/em> file add the plugin along with two more dependencies which will be used to render the UI icons later for our app as shown below.<\/p>\n\n\n\n
Update settings.gradle file as the following to link openCV and Vector Icons.<\/p>\n\n\n\n
..\ninclude ':app'\n \ninclude ':react-native-vector-icons'\nproject(':react-native-vector-icons').projectDir = new File(rootProject.projectDir,\n '..\/node_modules\/react-native-vector-icons\/android')\n \ninclude ':openCVLibrary310'\nproject(':openCVLibrary310').projectDir = new File(rootProject.projectDir,'..\/node_modules\/react-native-rectangle-scanne\nr\/android\/openCVLibrary310')\n\n<\/code><\/pre>\n\n\n\n
Diving into Code<\/strong><\/p>\n\n\n\n
We will begin by initializing a stateful React Component and define the propTypes<\/em> for the props that we will be using later and initialize the state variables. We then utilize the createRef provided by React to initialize a reference and assign it to the camera variable, which will be later used for camera actions.<\/p>\n\n\n\n
import { PropTypes } from 'prop-types';\n \nexport default class DocumentScanner extends React.Component {\n static propTypes = {\n cameraIsOn: PropTypes.bool,\n onLayout: PropTypes.func,\n onPictureTaken: PropTypes.func,\n onPictureProcessed: PropTypes.func\n }\n \n static defaultProps = {\n cameraIsOn: undefined, \/\/ Whether camera is on or off\n onLayout: () => { }, \/\/ Invokes when the camera layout is initialized\n onPictureTaken: () => { }, \/\/ Invokes when the picture is taken\n onPictureProcessed: () => { } \/\/ Invokes when the picture is taken and cached.\n }\n \n constructor(props) {\n super(props);\n this.state = {\n flashEnabled: false,\n showScannerView: false,\n didLoadInitialLayout: false,\n detectedRectangle: false,\n isMultiTasking: false,\n loadingCamera: true,\n processingImage: false,\n takingPicture: false,\n overlayFlashOpacity: new Animated.Value(0),\n device: {\n initialized: false,\n hasCamera: false,\n permissionToUseCamera: false,\n flashIsAvailable: false,\n previewHeightPercent: 1,\n previewWidthPercent: 1,\n },\n };\n \n this.camera = React.createRef();\n this.imageProcessorTimeout = null;\n }\n..\n<\/code><\/pre>\n\n\n\n
We define a function called onDeviceSetup() which basically retrieves information from the platform such as whether the device permissions for the camera are set, the aspect ratio when the preview is generated, etc.<\/p>\n\n\n\n
To notify various errors that may arise when accessing the camera getCameraDisabledMessage() is used.<\/p>\n\n\n\n
getCameraDisabledMessage() {\n if (this.state.isMultiTasking) {\n return 'Camera is not allowed in multi tasking mode.';\n }\n \n const { device } = this.state;\n if (device.initialized) {\n if (!device.hasCamera) {\n return 'Could not find a camera on the device.';\n }\n if (!device.permissionToUseCamera) {\n return 'Permission to use camera has not been granted.';\n }\n }\n return 'Failed to set up the camera.';\n }\n<\/code><\/pre>\n\n\n\n
Create a function turnOnCamera() which will open the scanner view. The turnOffCamera() function similarly hides the camera view, and also if the camera view is on but no camera was found in the device after calling the onDeviceSetup() method, it can optionally uninitialize the camera. The turnOnCamera() will be called each time immediately after a view update occurs.<\/p>\n\n\n\n
The turnOnCamera() and turnOffCamera() methods are invoked using lifecycle methods.<\/p>\n\n\n\n
The camera will be turned on inside componentDidMount() only after loading the initial layout when multi-tasking mode is off on iOS devices. Otherwise, the turnOffCamera() function will get invoked from componentDidUpdate(). Also, the imageProcessorTimeout timer which will be set at the time of capture failure should be cleared inside the componentWillUnmount() function.<\/p>\n\n\n\n
componentDidMount() {\n if (this.state.didLoadInitialLayout && !this.state.isMultiTasking) {\n this.turnOnCamera();\n }\n }\n \n componentDidUpdate() {\n if (this.state.didLoadInitialLayout) {\n if (this.state.isMultiTasking) return this.turnOffCamera(true);\n if (this.state.device.initialized) {\n if (!this.state.device.hasCamera) return this.turnOffCamera();\n if (!this.state.device.permissionToUseCamera) return this.turnOffCamera();\n }\n if (this.props.cameraIsOn === true && !this.state.showScannerView) {\n return this.turnOnCamera();\n }\n if (this.props.cameraIsOn === false && this.state.showScannerView) {\n return this.turnOffCamera(true);\n }\n if (this.props.cameraIsOn === undefined) {\n return this.turnOnCamera();\n }\n }\n return null;\n }\n \n componentWillUnmount() {\n clearTimeout(this.imageProcessorTimeout);\n }\n<\/code><\/pre>\n\n\n\n
In some Android devices, the aspect ratio of the preview is different than the screen size which may lead to a distorted camera preview. To deal with this issue write a utility function getPreviewSize()which will take the device height and width into account and return an appropriate preview size for the same.<\/p>\n\n\n\n
getPreviewSize() {\n const dimensions = Dimensions.get('window');\n \/\/ We use set margin amounts because for some reasons the percentage values don't align the camera preview in the center correctly.\n const heightMargin = (1 - this.state.device.previewHeightPercent) * dimensions.height \/ 2;\n const widthMargin = (1 - this.state.device.previewWidthPercent) * dimensions.width \/ 2;\n if (dimensions.height > dimensions.width) {\n \/\/ Portrait\n return {\n height: this.state.device.previewHeightPercent,\n width: this.state.device.previewWidthPercent,\n marginTop: heightMargin,\n marginLeft: widthMargin,\n };\n }\n \/\/ Landscape\n return {\n width: this.state.device.previewHeightPercent,\n height: this.state.device.previewWidthPercent,\n marginTop: widthMargin,\n marginLeft: heightMargin,\n };\n }\n \n<\/code><\/pre>\n\n\n\n
The function triggerSnapAnimation() is used to show the flash animation when the user captures an image.<\/p>\n\n\n\n
The capture() function is used to capture the current frame or the identified rectangle region. The loading or processing state will be set at the time of capturing to prevent any further capture triggers.<\/p>\n\n\n\n
capture = () => {\n if (this.state.takingPicture) return;\n if (this.state.processingImage) return;\n this.setState({ takingPicture: true, processingImage: true });\n this.camera.current.capture();\n this.triggerSnapAnimation();\n \n \/\/ If capture failed, allow for additional captures\n this.imageProcessorTimeout = setTimeout(() => {\n if (this.state.takingPicture) {\n this.setState({ takingPicture: false });\n }\n }, 100);\n }\n<\/code><\/pre>\n\n\n\n
We will be using the props provided by the plugin to process and cache the image as shown below. Here the image state will be set with the cached image url to be used in the preview.<\/p>\n\n\n\n
\/\/ The picture was captured but still needs to be processed.\n onPictureTaken = (event) => {\n this.setState({ takingPicture: false });\n this.props.onPictureTaken(event);\n }\n \n \/\/ The picture was taken and cached. You can now go on to using it.\n onPictureProcessed = (event) => {\n this.props.onPictureProcessed(event);\n this.setState({\n image: event,\n takingPicture: false,\n processingImage: false,\n showScannerView: this.props.cameraIsOn || false,\n });\n }\n<\/code><\/pre>\n\n\n\n
Depending on whether the device has a flashlight or not, the renderFlashControl() function will return a flash icon. To use the icons for the UI we will have to import vector icons as follows.<\/p>\n\n\n\n
import Icon from 'react-native-vector-icons\/Ionicons';<\/code><\/pre>\n\n\n\n
The renderCameraView() function is rendering either the camera view or a loading state, depending or an error message. Here the allowDetection prop is set true which allows automatic detection of an identifiable rectangular region and then triggers the onDetectedCapture prop where we extract and process the detected document.<\/p>\n\n\n\n
You have to import the Scanner and RectangleOverlay from the react-native-rectangle-scanner package.<\/p>\n\n\n\n
Now we can piece this all together to render the final UI. Here, if the image state is set, we will be redirected to the preview page and if not, the camera view will be rendered from which we can capture the image.<\/p>\n\n\n\n
render() {\n if (this.state.image) {\n return (\n <View style={styles.previewContainer}>\n <View style={styles.previewBox}>\n <Image source={{ uri: this.state.image.croppedImage }} style={styles.preview} \/>\n <\/View>\n <TouchableOpacity style={styles.buttonContainer} onPress={this.retryCapture}>\n <Text style={styles.buttonText}>Retry<\/Text>\n <\/TouchableOpacity>\n <\/View>\n )\n } else {\n return (\n <View\n style={styles.container}\n onLayout={(event) => {\n \/\/ This is used to detect multi tasking mode on iOS\/iPad\n \/\/ Camera use is not allowed\n this.props.onLayout(event);\n if (this.state.didLoadInitialLayout && Platform.OS === 'ios') {\n const screenWidth = Dimensions.get('screen').width;\n const isMultiTasking = (\n Math.round(event.nativeEvent.layout.width) < Math.round(screenWidth)\n );\n if (isMultiTasking) {\n this.setState({ isMultiTasking: true, loadingCamera: false });\n } else {\n this.setState({ isMultiTasking: false });\n }\n } else {\n this.setState({ didLoadInitialLayout: true });\n }\n }}\n >\n <StatusBar backgroundColor=\"black\" barStyle=\"light-content\" hidden={Platform.OS !== 'android'} \/>\n {this.renderCameraView()}\n <\/View>\n );\n }\n }\n \n retryCapture = () => {\n this.setState({\n image: null\n });\n }\n<\/code><\/pre>\n\n\n\n
You might have noticed that there are a lot of custom style references which scaffolds out the views properly which is defined below.<\/p>\n\n\n\n
![\"\"](\"https:\/\/lh3.googleusercontent.com\/cOB4MLBnUZtmflXeZ38yvgX5EFsP-HEO4FAeIBhb4EKQaieV4elg_G0PaAeBNxSQGY1d8rhgkL9d7CuTpWynusaux1tosLjc8YDfyUMb9jDxXu6zP_MLcGJA5OPqXezefrIWgRIv03zOKXUfzA\")
<\/figure><\/div>\n\n\n\n